Inside Rafter

Journal: Building Commands Integration

This is heavily inspired by Laravel Vapor, where you can fire off a command to a worker service. It hooks into framework-specific CLI utilities like php artisan for Laravel, rails for Rails, and wp for WordPress.

April 28, 2020

• Decided that Rafter will expose a Commands UI that allows a user to create a new Command model, which is then dispatched via HTTP to the worker service
• Did lots of research on how to properly authenticate the request to the non-public worker service. If you’re using a GCP service like Cloud Tasks or PubSub, it’s super easy because Google handles authenticating with the receiving service just fine. However, we need to call this synchronous request ourselves and consume the response to display it to the user. This means we need to sign the request ourselves with an OIDC token.
• Found a useful code snippet in the GCP docs for their IAP service (which we aren’t actually using). But it demonstrates how you can leverage the Google-provided ApplicationDefaultCredentials class to take care of all the messy JWT signing. All you have to do is pass an audience parameter (which, in this case, is the URL of the worker Cloud Run service), and you get an OIDC token ✨.

Let’s start with the Laravel Rafter Core integration, since that’s the easiest. In the service provider for the Laravel package, I register the worker-specific routes:

Route::group(['middleware' => [VerifyGoogleOidcToken::class, EnsureRafterWorker::class]], function () {
    Route::post(Rafter::QUEUE_ROUTE, 'Rafter\Http\Controllers\RafterQueueWorkerController');
    Route::post(Rafter::SCHEDULE_ROUTE, 'Rafter\Http\Controllers\RafterScheduleRunController');
});

• The VerifyGoogleOidcToken middleware is custom-built to verify an incoming request contains the proper Authorization header…
• And now that I type all of this out, I realize this probably isn’t even necessary 🙈 since Google Cloud Run already authenticates the request. D’oh.

Anyway, carrying on. I’m adding a new route to the Rafter class:

class Rafter
{
    const QUEUE_ROUTE = '/_rafter/queue/work';
    const SCHEDULE_ROUTE = '/_rafter/schedule/run';
    const COMMAND_ROUTE = '/_rafter/command/run';

As well as a new route in the service provider, pointing to a new RafterCommandRunController:

namespace Rafter\Http\Controllers;

use Illuminate\Http\Request;
use Illuminate\Routing\Controller;
use Illuminate\Support\Facades\Artisan;

class RafterCommandRunController extends Controller
{
    public function __invoke(Request $request)
    {
        Artisan::call($request->command);
        return Artisan::output();
    }
}

It’s pretty straightforward:

• It uses the Artisan Facade to call a given command that is passed in through a request parameter
• Then it returns the output of the command, also using the Artisan facade.

Time to test this baby out! I have a Rafter Laravel Example project locally that I’ll use.

• First, I’ll borrow Caleb Porzio’s composer-link bash alias to mount the local package into the project:

composer-link ../laravel-rafter-core

• Update: This didn’t work 🙃 because I’m already referencing a package published on Packagist.
• Instead, I had to add the git repository and reference dev-{branch} e.g. dev-commands as the version name, and then run composer update. Thanks to this blog post!

And my curl test worked great 🎉:

curl -d '{"command": "list"}' -H "Content-Type: application/json" -X POST http://rafter-example-laravel.test/_rafter/command/run
Laravel Framework 7.9.2

Usage:
  command [options] [arguments]

Options:
  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
      --env[=ENV]       The environment the command should run under
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
  clear-compiled       Remove the compiled class file
  ...

⚡️ I’ve opened a PR for the Laravel Core Package here.

Next, let’s focus on the Rafter integration. This will be a little more work:

• Let’s have fun with some UI-Driven Development. How do we want the Commands UI to look?
• I’m sure we’ll want to add a tab under the environments.show partial:

• This is a simple thing to add, but where do we point? Probably something like projects.environments.commands.index? I’m really not digging these nested paths. I’m not married to this setup. I’m also not a huge fan of having to create a controller and a view just to render a couple lines of a component, probably Livewire.
• Livewire author Caleb Porzio demonstrated a Route helper in a recent screencast. I wonder if I can start taking advantage of this, and use Livewire primarily for these small settings screens and tabs without having to use controllers?
• Or maybe I just use anonymous, inline functions instead?

Meh, I’ll do an inline function:

// Commands
Route::get('projects/{project}/environments/{environment}/commands', function (\App\Project $project, \App\Environment $environment) {
    return view('environments.commands', [
        'project' => $project,
        'environment' => $environment,
    ]);
})
    ->name('projects.environments.commands.index');

I hate it already. Onward!

• Let’s mock up what we want the commands index to look like. Probably a table of some sort, similar to our deployments list table.
• WAIT… this probably needs to be paginated. And I would be remiss if we built this without using Livewire’s fancy AJAX pagination. Let’s just make CommandsList a Livewire component 😈

php artisan make:livewire CommandsList

Then let’s try simplifying the route:

Route::livewire('projects/{project}/environments/{environment}/commands', 'commands-list')
    ->layout('components.environment')
    ->name('projects.environments.commands.index');

• Ugh. Update: this approach requires you to use a traditional Laravel layout file with stuff like @yield('content'). I’m using a newer component-based layout approach, using {{ $slot }}.
• I’m just gonna switch back to using a normal view, which renders a Livewire component.

Here’s environments/commands.blade.php

<x-layout>
  <x-environment :project="$project" :environment="$environment">
    <x-subtitle>Commands</x-subtitle>

    <livewire:commands-list :environment="$environment" />
  </x-environment>
</x-layout>

And the CommandsList.php Livewire component:

<?php

namespace App\Http\Livewire;

use App\Environment;
use Livewire\Component;

class CommandsList extends Component
{
    public $environment;

    public function mount(Environment $environment)
    {
        $this->environment = $environment;
    }

    public function render()
    {
        return view('livewire.commands-list');
    }
}

And here’s some inital UI, stubbed out with fake data:

I dig it so far!

• Next, let’s work on the new command view.
• Again, I’m torn on how I want this to work.
• It could be a simple Blade template with a form which POSTs to a controller, and redirects the user to a show page with the results.
• But I want this to feel interactive, as in: when you click Run, you see some sort of a spinner

Several hours later…

• OK I originally planned to make this a synchronous action, shying away from the notion that I might make this an asynchronous DispatchCommand job.
• However, making it an async job makes my UI decision a bit easier: I’ll create the command, dispatch it onto a queue, and immediately redirect the user to the commands.show page. This will be another Livewire component which polls for updates to the output.
• Plus, you have to think that sometime a user is going to run a super lengthy artisan command. We want Rafter to feel fast even if their worker service is slow.

OK - with these plans in mind, let’s start in on a new view for New Commands.

• I think I’m gonna make it a Livewire component for the hell of it. Might add some cool functionality later on, like autocompleting commands based on project, etc. Plus this is going to contain some logic, e.g. prepending the php artisan for the user alongside the input.
• ~Adding a new view and route for commands.show.~ NOPE… let’s just add the input to the top of commands.index for simplicity.

At the top of my existing CommandsList Livewire component, I’ve added a new form:

<div class="mb-4">
  <form wire:submit.prevent="runCommand">
    <label for="command" class="sr-only">
      Run a new command
    </label>
    <div class="mt-1 flex rounded-l-md shadow-sm">
      <span
        class="inline-flex items-center px-3 rounded-l-md border border-r-0 border-gray-300 bg-gray-50 text-gray-500 border-r-0"
      >
        php artisan
      </span>
      <input
        wire:model="newCommand"
        id="command"
        class="form-input flex-1 block w-full px-3 py-2 rounded-none sm:leading-5 autofocus"
        placeholder="command"
      />
      <span class="inline-flex rounded-r-md shadow-sm">
        <button
          type="submit"
          class="inline-flex items-center px-4 py-2 border border-transparent leading-5 font-medium rounded-r-md text-white bg-indigo-600 hover:bg-indigo-500 focus:outline-none focus:border-indigo-700 focus:shadow-outline-indigo active:bg-indigo-700 transition ease-in-out duration-150"
        >
          Run
        </button>
      </span>
    </div>
  </form>
</div>

Looks pretty spiffy:

• Now to actually wire this thing up.
• This will be an actual controller route and Blade view - handy for being able to link to existing commands.
• Let’s go ahead and create the Command model, controller, and other fun stuff:

php artisan make:model Command -a

Here’s my migration:

Schema::create('commands', function (Blueprint $table) {
    $table->id();
    $table->unsignedBigInteger('environment_id')->index();
    $table->unsignedBigInteger('user_id');
    $table->string('command');
    $table->longText('output')->nullable();
    $table->string('status')->default('pending');
    $table->timestamps();

    $table->foreign('environment_id')
        ->references('id')->on('environments')
        ->onDelete('cascade');
});

And let’s generate a CommandOutput Livewire component:

php artisan make:livewire CommandOutput

CommandOutput.php is going to accept the Command as its only argument:

<?php

namespace App\Http\Livewire;

use App\Command;
use Livewire\Component;

class CommandOutput extends Component
{
    public $command;

    public function mount(Command $command)
    {
        $this->command = $command;
    }

    public function render()
    {
        return view('livewire.command-output');
    }
}

Let’s add a view, environments.commands.show, to house the new component. Something like this:

<x-layout>
  <x-environment :project="$project" :environment="$environment">
    <x-subtitle>Command: <code>{{ $command->command }}</code></x-subtitle>

    <livewire:command-output :command="$command" />
  </x-environment>
</x-layout>

Now let’s chat about the flow for actually creating and dispatching a command:

1. User enters new command in the index view and clicks the Run button (or presses Enter).
2. Livewire fires the runCommand method, which creates a new Command model, and fires the $command->dispatch() method on the model.
3. Livewire redirects the user to commands.show which will show the CommandOutput component. It will poll for the status and display the output when ready.
4. The model then marks itself as pending with $this->markPending(), and dispatches a new DispatchCommand job.
5. Inside the job, a signed request is made to the worker URL with the command payload of the current Command.
6. The response is returned and recorded in $command->update([ 'output' => $responseBodyText ])
7. The command is marked has having succeeded with $command->markFinished(). We also want to handle any error cases, designating with $command->markFailed().
8. The output is automatically displayed to the user inside the Livewire component.

So all of these things are pretty straightforward, except for this:

Inside the job, a signed request is made to the worker URL with the command payload of the current Command.

It’s gonna involve several moving pieces. Let’s start from the top, though.

Moving back to our creation form, let’s define some logic inside the runCommand method:

public function runCommand()
{
    $this->authorize('update', $this->environment);

    $this->validate([
        'command' => ['required', 'string'],
    ]);

    $command = $this->environment->commands()->create([
        'command' => $this->command,
        'user_id' => request()->user()->id,
    ]);

    $command->dispatch();

    return redirect()->route('projects.environments.commands.show', [
        $this->environment->project,
        $this->environment,
        $command
    ]);
}

Gosh I really dislike these nested routes. I might change them sometime to be flat.

Cool. Time to run my migrations and… try this out?

💥 Issue: Mass-assignment. Sigh.

Let’s update our Command model to remove mass-assignment protection and add some helper methods while we’re at it:

<?php

namespace App;

use Illuminate\Database\Eloquent\Model;

class Command extends Model
{
    const STATUS_PENDING = 'pending';
    const STATUS_RUNNING = 'running';
    const STATUS_FINISHED = 'finished';
    const STATUS_FAILED = 'failed';

    protected $guarded = [];

    public function dispatch()
    {
        # code...
    }

    public function markRunning()
    {
        $this->update(['status' => static::STATUS_RUNNING]);
    }

    public function markFinished(string $output)
    {
        $this->update([
            'status' => static::STATUS_FINISHED,
            'output' => $output,
        ]);
    }

    public function markFailed(string $output)
    {
        $this->update([
            'status' => static::STATUS_FAILED,
            'output' => $output,
        ]);
    }
}

✨OK my redirect is working at least. I’ll move on to new steps tomorrow 🌄.

April 29, 2020

Good morning! Let’s keep going.

• I think I’ll stub out a command.show layout, since right now it’s blank.
• Probably just show the command name, when it ran, its current status, and below we can print out the results.

<div>
  <div class="flex justify-between items-center mb-4">
    <span>
      {{ $this->label }} {{ $command->updated_at->diffForHumans() }} @if
      ($command->isFinished()) - {{ $command->elapsedTime() }} @endif
    </span>
    <span>
      <x-status :status="$command->status" />
    </span>
  </div>
  <div class="font-mono p-4 text-sm bg-white">{{ $command->output }}</div>
</div>

• I added some new logic in the component itself to provided a computed property label so we can say “Ran 10s ago” or “Started running 1s ago”, etc

public function getLabelProperty(): string
{
    if ($this->command->isRunning()) {
        return 'Started running';
    } elseif ($this->command->isFinished()) {
        return 'Ran';
    } elseif ($this->command->isFailed()) {
        return 'Failed';
    }

    return 'Created';
}

• I also added a $command->elapsedTime() to get the total runtime of the command.
• No idea whether this works.

public function elapsedTime(): string
{
    return $this->updated_at->longAbsoluteDiffForHumans($this->created_at);
}

• Now that we have some UI, let’s make it work behind the scenes!
• I’m going to create a job and the logic to call the worker service next.

🎉 I did a livestream of the next part! You can watch it on YouTube.

So I created a new method on the Command model:

public function runCommandOnWorker(): string
{
    $workerUrl = $this->environment->worker_url . '/_rafter/command/run';
    $jsonKey = $this->environment->project->googleProject->service_account_json;

    /**
     * Google helps us out by creating a middleware to sign the outgoing request to the
     * worker service with an OIDC token based on the audience (which is the $workerUrl).
    */
    $creds = new ServiceAccountCredentials(null, $jsonKey, null, $workerUrl);
    $middleware = new AuthTokenMiddleware($creds);

    $stack = HandlerStack::create();
    $stack->push($middleware);

    $client = new Client([
        'handler' => $stack,
        'auth' => 'google_auth'
    ]);

    try {
        $response = $client->post($workerUrl, [
            'form_params' => [
                'command' => $this->command,
            ],
        ]);

        $output = $response->getBody()->getContents();

        $this->markFinished($output);

        return $output;
    } catch (Exception $e) {
        $this->markFailed($e->getMessage());
    }
}

• This is kind of a long method, but it works so far.
• I might extract it out into a separate library, like CallMethodOnWorker, once I need to use it in multiple places.
• I confirmed that it works as expected on the console!

• Our final step in all of this is to extract it to an async job. Let’s go ahead and do that:

php artisan make:job DispatchCommand

Here’s what that looks like:

<?php

namespace App\Jobs;

use App\Command;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class DispatchCommand implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $command;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(Command $command)
    {
        $this->command = $command;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $this->command->markRunning();

        $this->command->runCommandOnWorker();
    }
}

And here’s how I’m dispatching it inside Command:

public function dispatch()
{
    DispatchCommand::dispatch($this);
}

Let’s… see how it works? I’m gonna record the first attempt. No polish on this: you’ll see the real thing!

🎉🎉🎉

Nice. I had to manually refresh, because I forgot I haven’t added live polling yet.

Let’s go ahead and add live polling by adding wire:poll to the root of the component:

<div wire:poll>
  <div class="flex justify-between items-center mb-4">
    <span>
      {{ $this->label }} {{ $command->updated_at->diffForHumans() }} @if
      ($command->isFinished()) - {{ $command->elapsedTime() }} @endif
    </span>
    <span>
      <x-status :status="$command->status" />
    </span>
  </div>
  <div class="font-mono p-4 text-sm bg-white">
    <pre>{{ $command->output }}</pre>
  </div>
</div>

Easy as that!

• I’ve done some redesigning. Not gonna talk about it in-depth, but I feel better about the way things look.
• I wish there were a way to conditionally trigger Livewire polling. This is probably possible, and I might add a PR for it. BTW I tried conditionally adding livewire:poll using Blade directives, but it looks like the component keeps trying to fetch a new partial at the poll interval after it’s mounted, regardless of whether the new partial has the poll directive.

⚠️ I’ve now come across some things we should tidy up:

• Failed commands inside laravel-rafter-core result in 500 errors. We probably do not want this; rather, we want to catch the errors and return them in a 200 response.
• We need to add some authorizations to a CommandPolicy to ensure rando users cannot view other users’ commands.
• I wonder if we can stream Artisan responses back to Rafter and update the output as the stream comes back?

Let’s start with the command cleanup. Inside the RafterCommandRunController:

public function __invoke(Request $request)
{
    try {
        Artisan::call($request->command);
        return Artisan::output();
    } catch (Exception $e) {
        return $e->getMessage();
    }
}

• Hmm this works, but should it really be a 200 response? Should we use some sort of 4xx or 5xx status code? I wonder if it will feel odd to see a command as green “Finished” but have an error output.
• Meh I’ll stick with 200 for now.
• This is better!

April 30, 2020

Now, let’s add a policy and a test to authorize the Command routes:

php artisan make:test CommandControllerTest

In this test, we’ll create a few commands for a given user’s environment, and then an additional command that doesn’t belong to the user. We’ll assert that the user can only see their commands:

namespace Tests\Feature;

use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Foundation\Testing\WithFaker;
use Tests\TestCase;

class CommandControllerTest extends TestCase
{
    use RefreshDatabase;

    protected $user;
    protected $other;

    public function setUp(): void
    {
        parent::setUp();

        $this->user = factory('App\User')->create();
        $this->other = factory('App\Command')->create();
    }

    public function test_user_can_view_their_commands()
    {
        $project = factory('App\Project')->create([
            'team_id' => $this->user->currentTeam->id,
        ]);

        $environment = factory('App\Environment')->create([
            'project_id' => $project->id,
        ]);

        $commands = factory('App\Command', 3)->create([
            'environment_id' => $environment->id,
            'user_id' => $this->user->id,
        ]);

        // Also ensure command created by another user on the same team is visible
        $otherTeamMember = factory('App\User')->create();
        $this->user->currentTeam->users()->attach($otherTeamMember);

        $otherCommandOnTeam = factory('App\Command')->create([
            'environment_id' => $environment->id,
            'user_id' => $otherTeamMember->id,
        ]);

        $this->user->currentTeam->refresh();

        $response = $this->actingAs($this->user)
            ->get(route('projects.environments.commands.index', [$project, $environment]))
            ->assertSuccessful();

        $commands->each(function ($command) use ($response) {
            $response->assertSee($command->command);
        });

        $response->assertSee($otherCommandOnTeam->command);

        $response->assertDontSee($this->other->command);

        $this->get(route('projects.environments.commands.show', [$project, $environment, $commands->first()]))
            ->assertSuccessful();

        $this->get(route('projects.environments.commands.show', [
            $this->other->environment->project,
            $this->other->environment,
            $this->other
        ]))
            ->assertForbidden();
    }
}

Of course, we’ll need to update the Command factory with some real stuff first:

use App\Command;
use Faker\Generator as Faker;

$factory->define(Command::class, function (Faker $faker) {
    return [
        'command' => 'route:list',
        'user_id' => factory('App\User'),
        'environment_id' => factory('App\Environment'),
        'status' => 'pending',
    ];
});

Now let’s run the test and see what we get!

We got one failing test, here:

$ php artisan test

There was 1 failure:

1) Tests\Feature\CommandControllerTest::test_user_can_view_their_commands
Response status code [200] is not a forbidden status code.
Failed asserting that false is true.

This means that the user was able to view a different user’s commands. Let’s fix that using a Policy:

php artisan make:policy CommandPolicy -m Command

And let’s define a policy for the viewAny and view methods, since those will correspond to the index and show controller methods:

/**
 * Determine whether the user can view any commands.
 *
 * @param  \App\User  $user
 * @return mixed
 */
public function viewAny(User $user)
{
    return true;
}

/**
 * Determine whether the user can view the command.
 *
 * @param  \App\User  $user
 * @param  \App\Command  $command
 * @return mixed
 */
public function view(User $user, Command $command)
{
    return $user->currentTeam->is($command->environment->project->team);
}

Finally, let’s instruct the CommandController to authorize its methods against a policy in the constructor:

public function __construct()
{
    $this->authorizeResource('App\Command');
}

Cool! Running the test again… and everything passes! 🎉

May 4, 2020

One final thing I want to do before shipping: DRY up the php artisan prefix.

• This will change per-project: rails or wp etc.
• I’m also considering finding a way to execute a shell command directly, giving the user total access to the shell, rather than force them to go through a hardcoded utility library.
• But for now, let’s find a place to start the “command prefix” for Laravel projects, and pull it from there. In Project.php:

/**
 * Get the prefix for running Commands for a given project type.
 *
 * @return string
 */
public function commandPrefix(): string
{
    if ($this->type == 'laravel') {
        return 'php artisan';
    }

    return '';
}

• Good enough for now - I’ll probably revisit this and make some sort of abstraction for ProjectType eventually.
• Finally, not all project types are going to have Command capabilities (at least to start)! Let’s ensure we only show the Command UI for certain project types. Again, in Project:

public function usesCommands(): bool
{
    return $this->type == 'laravel';
}

• Again, super basic, and will likely be extracted in the future.
• I’m also flirting with the idea that I should add a check at the controller-level to ensure someone can’t view or create new commands for projects that can’t run them. This is definitely something I should add eventually, but I’m not too concerned about it yet. I’ll add an issue for it.

✨ This was shipped! ✨

Thanks for following along 😀 I’ll try to write up a proper blog post soon.

A blog about building Rafter, a serverless deployment platform. Posts by Josh Larson. Follow Josh on Twitter for updates.

• ← What is Rafter?
• Tracking Queued Job Chains in Laravel →