Build a Node.js runner script to run a Linux program in the background

Node.js Runner

Description:

  A script that listens for a PostgreSQL event "newJob" and uses the event JSON to schedule and spawn the dedworker program, later updating a record of the program running (job) to indicate an error, if it's running or complete.

Technologies:
  - Nodejs v4+
    - child_process spawn
    - postgres pub sub
  - PostgreSQL 9+

Explination of why this is needed:

  The dedworker runs on the Linux machine to do things like cache contacts, scan for duplicates, etc.
  Since the dedworker is just a program that runs and ends, it doesn't know how to start itself.
  This is where a Node.js Runner needs to spawn the dedworker program when it's needed.

  We schedule dedworker to be run by Nodejs by inserting a "job" into postgres, allowing Nodejs Runner to find it, and use the information to spawn the dedworker program.

  For example, we want to run dedworker to fetch contacts from a client's CRM every hour. So everytime dedworker runs, it will insert a job to the postgres table "jobs" and emit the event "newJob" with a JSON { jobid: 123, job: "somename", userid: 123 } so that Node.js Runner can schedule it to run when the job.schedule_time is set.

  We need to know when a job has an error, when it started and when it finished. Nodejs can catch this when spawning dedworker and make the update in the database.

Detalles del Mini-Projecto:

  - Listen for PostgreSQL "newJob" event. Please use a .json config for storging and fetching the pg connection info.
 
  - When a "newJob" event is triggered, check the postgres job table to make sure a job by that name and userid is not already have status "RUNNING". (We don't want to run multiple dedworker jobs for the same user at the same time)

  - Get the job from postgres by using the "jobid" passed from the event JSON.
 
  - Spawn "dedworker" using the JSON keys from the event { "job": "jobName", "jobid": 123, "userid": 123 } to pass as parameters to the dedworker script for arguments --job, --jobid, --userid
 
  - If the job.schedule_time is in the future, queue it using setTimeout to run in the future
 
  - After spawning, update the "job" entry in postgres(using jobid to reference the Postgres job.id), the status to "running" and set job.start_time to current time.

  - If the spawned process ends in an error, update the postgres job entry to "error", if there is no job.error_message, add any error message
 
  - When the spawned process is completed, update the postgres job entry status to "complete" and update job.end_time

Here is an example postgres job table you can use for testing:

  create table "job"(
    "id" serial primary key,
    "schedule_time" bigint not null,
    "start_time" bigint,
    "end_time" bigint,
    "name" text not null,
    "user_id" int,
    "status" text not null default 'queued',
    "error_message" text    -- The error message that happened while running the actual job
  );

You can create a fake dedworker program to test spawning. For example: ./Dedworker.sh