(hopefully a compilation of)

working notes that cause SWIM procedures to work

==================== collection

jump kit

somehow I acquired Jumpstartkit-5.1.1.zip

editing

source file is at:

vim ./swim/java/JumpstartKit-5.1.1/JumpstartKit/
    /src/main/java/com/harris/gcsd/dex/jumpstart/CssiConsumer.java

setup

d/l ant from:

    https://ant.apache.org/bindownload.cgi

<don't have the installations, but it seems to be installed at:

    swim\java\apache-ant-1.10.5\

daily (regular) commands

BUILD: in dos window:

    C:\cygwin64\home\wturner\swim\java\JumpstartKit-5.1.1\JumpstartKit>
        ..\..\apache-ant-1.10.5\bin\ant.bat

BUILD: in cygwin window:

    WTurner@ASDI-DB ~/swim/java/JumpstartKit-5.1.1/JumpstartKit
        $ ../../apache-ant-1.10.5/bin/ant

RUN: in dos or cygwin:

    cd  ~/swim/java/JumpstartKit-5.1.1/JumpstartKit/bin
    runcssi.bat
        - or -
    doit.sh

may possibly run detached:

    cygstart doit.sh

subscriptions & queues

Below are the current endpoint subscriptions for the CSSI STDDS queues on the R&D Solace device.

CSSI.STDDS.Q01.OUT - STDDS - APDS RVR   << this doesn't seem right >>
CSSI.STDDS.Q02.OUT - STDDS - TAIS
CSSI.STDDS.Q03.OUT - STDDS - SMES ASDEX Position Reports, SMES ASDEX System
    Status, and SMES Surface Movement Events

We can filter out the Surface Movement Events from the CSSI.STDDS.Q03.OUT if needed.

config files

ant compilation:

JumpstartKit/build.xml

java run-time:

conf/consumerConnection.properties

TODO: split cssi postgres code from jumpstart mainline

TODO: log xml only if postgres insert fails

TODO: determine difference of Log(log4j) and Sys.print

done: merge code for Q01, Q02, Q03

database

postgres install

followed regular install BUT somewhere in the install make sure to check (somewhere) 'GIS additions' Don't remember if I got it from postgres.org or https://postgis.net/windows_downloads/

PROBLEM: now have both v9.6 and v10 on asdi-db!!!

s.b.: UNINSTALL v9.6 !!!

this file reflects the changes from v9.6 to v10

DONE. V10 is used, but v9 was NOT uninstalled

AND port not changed

d.b and table creation

psql runs from either dos or cygwin:

    "C:\Program Files\PostgreSQL\10\bin\psql.exe" -h localhost -p 5433 cssitest postgres
    -- or --
    > psql -h localhost -p 5433 swim adminw

webfaction:

    > psql cssitest adminw

postgres v10 / postgis 2.5

consider: if delete all & re-install, use:

postgres=# create database swim;
CREATE DATABASE

cssitest=# CREATE EXTENSION postgis;
CREATE EXTENSION

done

rvr

CREATE TABLE rvr(
    airport                varchar(4)   NOT NULL,
    runway                 varchar(4)   NOT NULL,
    rvrtime                timestamptz default current_timestamp,

    touchdownvisualrange   varchar(2),
    touchdowntrend         varchar(1),
    midpointvisualrange    varchar(2),
    midpointtrend          varchar(1),
    rolloutvisualrange     varchar(2),
    rollouttrend           varchar(2),
    runwayedgelightsetting varchar(1),
    runwaycenterlinelightsetting varchar(1)
);

fdps

CREATE TABLE fdps(
  acid           text        NOT NULL,
  msg_arrival    timestamptz NOT NULL, -- 2018-09-07T03:12:49.073Z
  gufi_fdps      text,                 --  us.fdps.2018-09-07T00:47:40Z.000/07/600
  gufi_hex       text,                 -- 9791a054-6b4a-41ad-a722-b6d0f09d4218
  dep_apt        text,
  arr_apt        text,

  dep_time       timestamptz, -- 2018-08-31T02:31:00Z
  arr_time       timestamptz, -- 2018-08-31T05:14:00Z

  fltplan_id     text,        -- KH02860600
  flt_status     text,        -- ACTIVE
  actype         text,        -- P28R

  position       geometry(Point, 4326),
  lat            float,
  lng            float,
  actualspeed    float(1),
  altitude       float(1),

  trkvel_x       float(1),
  trkvel_y       float(1)
);

stdds

CREATE TABLE stdds(

  src_airport  text      NOT NULL,   -- TODO: change this to 'src'!
  trackNum     integer   NOT NULL,
  acAddress    text,
  status       text,

  mrtTime      timestamptz NOT NULL, -- Q: NOT NULL ???

  lat          float,
  lon          float,
  position     geometry(Point, 4326),

  vVert        integer,
  vVertRaw     integer,
  vx           integer,
  vxRaw        integer,
  vy           integer,
  vyRaw        integer,

  xPos         integer,
  yPos         integer,

  adsb         integer,
  frozen       integer,

  new          integer,
  pseudo       integer,

  reportedAltitude   integer,
  reportedBeaconCode text
);

flightplan

CREATE TABLE flightplan(

  src_airport  text      NOT NULL,   -- TODO: change this to 'src'!
  trackNum     integer   NOT NULL,   -- saved from <track> record

  acid         text      NOT NULL,
  sfpn         integer,

  acType       text,
  airport      text,
  category     text,
  cps          text,
  dbi          text,
  ECID         text,
  entryFix     text,
  exitFix      text,
  flightRules  text,
  lld          text,
  ocr          text,
  ptdTime      text,
  rnav         text,
  runway       text,
  scratchPad1  text,
  scratchPad2  text,
  status       text,
  delete       integer,
  suspended    integer,
  type         text,

  requestedAltitude  integer,
  assignedBeaconCode text
  );

enhanced

added Oct 1: TODO: include the in with the flight plan ???
Q: how to tell when to do insert? or or
Q: or have this a separate table, with trackNum, and INSERT when ??

CREATE TABLE enhanced(

  src_airport          text      NOT NULL,   -- TODO: change this to 'src'!
  trackNum             integer   NOT NULL,   -- saved from <track> record

  eramgufi             text,
  sfdpsgufi            text,
  departureairport     text,
  destinationairport   text
  );

asdex

CREATE TABLE asdex(   -- or smesdata

  src        text          NOT NULL,      -- airport
  seqNum     integer       NOT NULL,
  time       timestamptz   NOT NULL,
  track      integer       NOT NULL,
  stid       integer       NOT NULL,

  latitude   float,
  longitude  float,
  position   geometry(Point, 4326),

  speed      integer,    -- CHANGED FROM FLOAT
  heading    integer,    -- CHANGED FROM FLOAT
  quality    integer,

  startRange    integer,   -- new for Wednesday 9/19
  endRange      integer,
  startAzimuth  float,
  endAzimuth    float,

  -- these rest occur when full="true":
  aircraftid text,
  tgttype    text,
  runway     text
);

maintenance

WTurner@ASDI-DB ~/swim/java/JumpstartKit-5.1.1/JumpstartKit/bin

OLD: psql -h localhost -p 5432 cssitest postgres
psql -h localhost -p 5433 swim postgres

Password for user postgres:

psql (10.5, server 9.6.5)

Type "help" for help.

simple queries:

view tables:
```
\dt
```
describe a table:
```
\d+ flightplandata
```

everything BUT position (that field is wide)

select src_airport, trackNum, acAddress, status, mrtTime, lat, lon, vVert,
    vVertRaw, vx, vxRaw, vy, vyRaw, xPos, yPos, adsb, frozen, new, pseudo,
    reportedAltitude, reportedBeaconCode
    from trackdata;

output to csv file:

 \copy (select * from trackdata) to 'lots_track.csv' csv header;
 \copy (select * from flightplandata) to 'lots_fp.csv' csv header;
 \copy (select * from asdexdata) to 'full_asdex_sep20.csv' csv header;

 \copy (select * from trackdata where src_airport='PCT' and
    mrttime >= '2018-09-19 00:00:00' limit 500000 ) to 'lots_track.csv' csv header;

quit:
```
 \q
```

input from csv file (MUST be 1,000,000 records or less):

cssitest=> \copy trackdata from 'lots_track.csv' delimiter ',' csv header;

insert into postgis from java:

    String wkt = "'POINT(" + sql_insert.get("lon") + ' ' +
                             sql_insert.get("lat") + ")'" ;
    pt = "ST_GeomFromText(" + wkt + ", 4326)";

    // e.g.: "ST_GeomFromText('POINT( -71.1234 38.4567)', 4326)";

delete everything (untested):
```
delete from rvrdata;
vacuum rvrdata;
```

or possibly:

    vacuum full rvrdata;

TODO: explain pgAdmin gui

python

-------- psycopg2

The psycopg2 could NOT be installed on cygwin python, so cygwin's python cannot be used, it installed in DOS python fine.

The psycopg2 installed fine on webfaction.

-------- dos vs. cygwin

need to run dos cygwin for web server so cgi-bin python can run via dos and do an import psycopg2

-------- web server

on cygwin home on asdi-db:

    cd httpd
    make copy      # copy useful files from 'npm run build' dist area
    make clean     # attempt to delete extra files, but THIS IS WRONG
    make run       # run python httpd server via dos

-------- cgi-bin

TODO: where is log file for cgi-bin???

TODO: add to json:

    properties: acid, altitude, heading
    point: last position (curpos, for symbol)
    metadata: date of query, time (length) of query, num
    tracks, etc.

ol5 (node)

-------- node installation

from: https://openlayers.org/en/latest/doc/tutorials/bundle.html

apparently, 'node' and other crap is needed...

node-v8.12.0-x64 d/l'ed and installed (typical inst)

Node.js has been successfully installed.

    > npm install ol

-------- node create project

WTurner@ASDI-DB

    $ cd ~/node
    $ mkdir cssi_first
    $ cd cssi_first/
    $ npm init

-------- config files

    cssi_first\package.json

==================== javascript

-------- edit

vim:

        ~/node/cssi_start/track.html
                     /tracking.js

TODO: delete old tracks

use xvel, yvel for heading

show altitude, radar data block

explore 'features:' in vectorSource instead of just ajax (XHR?) response

-------- run locally

npm start

-------- create distribution files

npm run build

which creates files:

~/node/cssi_start/dist/track.html
                      /tracking.<hexnumber>.js
                      /tracking.<hexnumber>.map

----------------- after one day

with no filters on stdds:

Enter input: d

rvr                850175   last: 10/16/2018 08:41 -04
fdps              5622665   last: 10/16/2018 07:49 -04
stdds            46722026   last: 10/16/2018 08:41 -04
flightplan       11522919
enhanced           835364
asdex            86689136   last: 10/16/2018 08:42 -04

and fdps barfs:

2018-10-16T11:49:57.444Z  UAL2148        null      null
2018-10-16T11:49:57.444Z  UAL2148        null      null
2018-10-16T11:49:57.472Z  DAL1478        null      null

+++++++++++++++++++++++++++++++++++++++++

NOT insert into whatever table ...

org.postgresql.util.PSQLException::ERROR: timestamp out of range: "20182026-07-08T13:07:00Z" Position: 235

+++++++++++++++++++++++++++++++++++++++++

and server/cgi seems way behind... stopped browser and server is still processing data

==================== asdi-db web server

use CherryPy!

==================== git

the git repo need to have ONLY sources, so find/install these packages:

not shown here:

    java/jdk
    postgres + postgis

----- how to install new cssi swim git repo, with packages

1) JumpstartKit

Harris JumpstartKit:

    https://www.faa.gov/air_traffic/technology/swim/documents/media/
    user-guide/JumpstartKit-5.1.1.zip

unzip only lib parts:

    unzip JumpstartKit-5.1.1.zip  'JumpstartKit-5.1.1/JumpstartKit/lib/*'

then move to proper location:

    mkdir blarf
    mv JumpstartKit-5.1.1/JumpstartKit/lib/ blarf/

2) Ant

    https://ant.apache.org/bindownload.cgi

install to ~/bin/ or ~/installs, and put that into $PATH

todo: edit build.xml for lib/CSSIfirst.jar to be something better

3) Node

ol5 instructions:

    https://openlayers.org/en/latest/doc/tutorials/bundle.html

actual loc of node:

    https://nodejs.org/en/

note: I think the:

    npm init

d/l's the entire package, so plan to do this accordingly

    npm install ol

4) Python

must use Python3 Windows version

    pip3.x install psycopg2
    pip3.x install cherrypy
    pip3.x install psutil

    pip3.x install subprocess
    pip3.x install pytz

11/30/2018:

how I think it works (as the rather lame html table):

rvr	enroute	terminal	surface
CSSI.STDDS.Q01.OUT	CSSI.FDPS.Q01.OUT	CSSI.STDDS.Q02.OUT	CSSI.STDDS.Q03.OUT
q01_Handler();	q_f01_Handler();	q02_Handler();	q03_Handler();
table: rvr	table: fdps	tables: trackdata, flightplandata, enhanceddata	table: asdex

same, but plain text:

rvr                | enroute           | terminal           | surface            |
-------------      | -------------     | -------------      | -------------      |
CSSI.STDDS.Q01.OUT | CSSI.FDPS.Q01.OUT | CSSI.STDDS.Q02.OUT | CSSI.STDDS.Q03.OUT |

q01_Handler();     | q_f01_Handler();  | q02_Handler();     | q03_Handler();     |

data/cssi_rvr      | data/cssi_fdps    | data/cssi_stdds_q02| data/cssi_asdex    |

table: rvr         | table: fdps       | tables: trackdata, | table: asdex       |
                   |                   |    flightplandata, |                    |
                   |                   |    enhanceddata    |                    |

the end.

================================================================= ======================== january 2019 ============================ =================================================================

updates for jan 2019

setup endpoint script to use table_dd format for all d.b. tables
created python script to stop, start all processes

scheduler

from: https://www.dummies.com/computers/pcs/how-to-open-windows-task-scheduler/

Control Panel -> System and Security -> Administrative Tools + Schedule Tasks
Actions -> Create Task
- Triggers: setup time & repeat
- Actions: + Action: Start a program
- Actions: + Program\script: C:\Users\wturner\Python37\python.exe
- Actions: + arguments: resequence.py
- Actions: + Start in: C:\cygwin64\home\wturner\swim2018\endpoint\cron

NOT SURE about any of it, will request help from Yasir or anyone who knows how to do it....

NOPE, cron task didn't work. Asked Yasir for help.

1/21/2018 attempts to use server-sent processing

Got server-sent to work locally from webapps

added srvrsent.ilc.com to cherrypy app
added srvrsent to site.py as needed
now this works: http://srvrsent.ilikecarrots.com/srvrsent.html

now have template for how server-sent operations work...

---------- server-sent receiver

receiving javascript is at:

        webapps/cherry/srvrsent.html

about the only difference is:

    var sse_handler = new EventSource('http://srvrsent.ilikecarrots.com/getUpdate');
    sse_handler.onmessage = function (event) {
        h_li.innerHTML = event.data;

that is, just use EventSource on the call to start

---------- server-sent sender

dns name created and added to cherry app
minor mods to site.py

main source is in:

        webapps/cherry/ss.py

"Content-Type" is "text/event-stream;charset=utf-8"
output contains "retry: 20000" which is interval to resent
output is particular about '\n' at end of lines and '\n\n' at end of json
output is json (use json.dumps(d) to output dictionary)

---------- refresher: how to update & run swin

build js code:

    cd ~/swim2018/node/cssi_tracker
    make dist

move 'compiled' code to server area

    cd ~/swim2018/cherry
    make copy

run server

    cd ~/swim2018/cherry
    make run
        C:\Users\WTurner\Python37\python.exe serve.py
        [21/Jan/2019:16:00:22] ENGINE Bus STARTING
        [21/Jan/2019:16:00:22] ENGINE Started monitor thread 'Autoreloader'.
        [21/Jan/2019:16:00:22] ENGINE Serving on http://0.0.0.0:8080
        [21/Jan/2019:16:00:22] ENGINE Bus STARTED

observe (from local firefox):

    http://asdi-db.cssiinc.com:8080/track.html

works as designed, tracks show up!

---------- proceeding with server-sent ops

1) edit js for EventSource
new url:

    var sse_handler = new EventSource('/get_tracks_srvrsent');

and in server.py: OLD:

    def get_tracks_ajax( self, apt, random):
        gjson = get_tracks.ajax( db_csr, "stdds", apt )

NEW:

    def get_tracks_srvrsent( self, apt ):
        gjson = get_tracks.srvrsent( db_csr, "stdds", apt )

after general updates, it works!

TODO: determine retry, 2000 doesn't seem to be 2 seconds
TODO: make datablocks useful
TODO: tracking across sources (stdds, asdex, fdps)
TODO: find out why fdps is inop
TODO: get asdex working

server-sent: GAVE UP.

this: https://cherrypydocrework.readthedocs.io/advanced.html#restful-style-dispatching says:

Streaming generators are sexy, but they play havoc with HTTP. CherryPy allows you to stream output for specific situations: pages which take many minutes to produce, or pages which need a portion of their content immediately output to the client. Because of the issues outlined above, it is usually better to flatten (buffer) content rather than stream content. Do otherwise only when the benefits of streaming outweigh the risks.

so, it appears that CherryPy doesn't do SSE very well.

and this: https://docs.cherrypy.org/en/latest/advanced.html#how-streaming-output-works-with-cherrypy

after several yield-generator attempts, it worked a little, but still seemed to close and open the connection on each message.

discussion on future of CherryPy

https://groups.google.com/forum/#!topic/cherrypy-users/lT1cxovGyy8

BACK to datablock attempts.

will try to focus on tracking and getting a good datablock

----- persistence of cherrypy data:

https://stackoverflow.com/questions/3898482/why-are-cherrypy-object-attributes-persistent-between-requests

To set data for the current session, use:

    cherrypy.session['fieldname'] = 'fieldvalue'

to get data use:

    cherrypy.session.get('fieldname')

Proposal:

Q: is cherrypy.session[] accessible from within get_tracks.py?
A: assume yes (for now)

candidates for persisitent data:

    cherrypy.session['mylg']     = mylg
    cherrypy.session['mydb']     = db
    cherrypy.session['day']      = day
    cherrypy.session['features'] = ???

cron job

====================== BAD DOS ATTEMPTS ======================

wmic:

C:\Users\wturner>wmic process where caption="python.exe" get  ProcessId
ProcessId
8604

and this:

C:\Users\wturner>wmic path win32_process get name, commandline
            sh.exe
C:\Users\wturner\Python37\python.exe serve.py

tasklist:

C:\Users\wturner>tasklist /FI "IMAGENAME eq python.exe"

Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
python.exe                    8604 RDP-Tcp#54                 2     29,992 K

and this (use %%# when inside .bat(?)):

C:\Users\wturner>FOR /F "tokens=2" %# in ('tasklist /fi "Imagename eq python.exe" ^| MORE +3') do (Echo %#)

C:\Users\wturner>(Echo 8604 )
8604

this looks promising:

C:\Users\wturner>wmic process where caption="python.exe" get commandline,processid
CommandLine                                    ProcessId
C:\Users\wturner\Python37\python.exe serve.py  8604

TOO HARD to do it in DOS, will use python!

====================== END of BAD DOS ATTEMPTS ======================

cron job (working)

re-wrote most everything in Python:

stop & restart java/swim/endpoint processes
stop cherrypy server
in .bat, use 'start /min' to start cherrypy server

2/20/2019

created script in endpoint/bin/create_tables.sh to create 01..31 tables for aa 6 of the tables
updated CssiConsumer.java to always use gmt day in tablename (on every INSERT statement
modified cgi/get_tracks.py to read gmt-day tablename; however, not sure if it really works

TODO:

update current (or start new) cron script to truncate (i.e., drop all date) from tables for tomorrow and the day after tomorrow
consider disabling (or turning off) the current java-recycle script

task manager

to access the cron scheduler:

Control Panel -> System and Security -> (Administrative Tools) Schedule Tasks

-> swim restart

current 'Action' value is:

Start a program ::: C:\Users\wturner\Python37\python.exe "C:\cyginw64\home\wturner\swim2018\endpoint\cron\resequence.py"

POSSIBLE truncate script (from https://www.davidjnice.com/cygwin_scheduled_tasks.html):

theirs:

c:\cygwin\bin\bash.exe -l -c "/full/path/to/command/plus-command >> /full/path/to/a/logfile 2>&1"

mine:

c:\cygwin64\bin\bash.exe -l -c "/home/WTurner/swim2018/endpoint/bin/truncate_tomorrow.sh >> /home/WTurner/swim2018/endpoint/bin/logfile_truncate.txt 2>&1"

looks good (runs from dos prompt)

Done. (created Task Scheduler item) -- NOT

REMOVED.

new plan: tablename_yyyymmdd

changed all d.b. tables to be xxx_yyyymmdd, using utc times
resequence.py does not delete anything
get_tracks.py seems to work fine using

new plan: 2/24/2019

problem: cron job does not run:
- used MyLog in resequence.py (was possibl problem) *now changed log file to C:/temp/

cherrypy as service:

some notes on trying to install :

pywin32 binaries: https://github.com/mhammond/pywin32/releases
pywin32 wheel: https://pypi.org/project/pywin32/#files
interesting text about cherrypy internals: https://github.com/cherrypy/cherrypy/wiki/CherryPySpec
source code printed out as python: https://github.com/cherrypy/tools/blob/master/WindowsService
more 3.0 and 3.1 attempts, with debugging tips: https://stackoverflow.com/questions/25741381/cherrypy-as-a-windows-service-the-example
nabble notes about redirecting stdout, stderr: http://python.6.x6.nabble.com/running-CherryPy-3-2-as-a-Windows-service-td1929705.html

note: wheel is installed via: pip install some-package.whl

1) windows (dos) python

I installed into my own local dir:

    python --version
    Python 3.7.0

    C:\Users\wturner>python -i
    Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

    where python
    C:\Users\wturner\Python37\python.exe

must use dos one since that version has:

        import psycopg2

2) cygwin python (hardly used):

    WTurner@ASDI-DB ~/swim2018
    $ python --version
    Python 2.7.14

3) uname -a

    CYGWIN_NT-6.3 ASDI-DB 2.11.1(0.329/5/3) 2018-09-05 10:24 x86_64 Cygwin

OK, will d/l this: location: https://github.com/mhammond/pywin32/releases file: pywin32-224.win-amd64-py3.7.exe into wturner/pkgs

=========== from the mhammond web page: ========== Installing via PIP

Note that PIP support is experimental.

You can install pywin32 via pip:

    pip install pywin32

Note that if you want to use pywin32 for "system wide" features, such as registering COM objects or implementing Windows Services, then you must run the following command from an elevated command prompt:

    python Scripts/pywin32_postinstall.py -install

SO, what the heck?

Ok, here goes the installation (clicked on file via 'file explorer')...

seems to have gone fine, here is the output:

    C:\Users\wturner\Python37\Scripts\pywin32_postinstall.py:156: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
        import imp
    Copied pythoncom37.dll to C:\Windows\system32\pythoncom37.dll
    Copied pywintypes37.dll to C:\Windows\system32\pywintypes37.dll
    Registered: Python.Interpreter
    Registered: Python.Dictionary
    Registered: Python
    -> Software\Python\PythonCore\3.7\Help[None]=None
    -> Software\Python\PythonCore\3.7\Help\Pythonwin Reference[None]='C:\\Users\\wturner\\Python37\\Lib\\site-packages\\PyWin32.chm'
    Pythonwin has been registered in context menu
    Creating directory C:\Users\wturner\Python37\Lib\site-packages\win32com\gen_py
    Shortcut for Pythonwin created
    Shortcut to documentation created
    The pywin32 extensions were successfully installed.

seems good so far...

Ok, just to test the install:

    WTurner@ASDI-DB ~/swim2018
    $ /cygdrive/c/Users/wturner/Python37/python.exe -i
    Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import win32serviceutil
    >>> import win32service
    >>>

will try the install mentioned at https://github.com/mhammond/pywin32

cygwin (not tried):
    cd /cygdrive/c/Users/wturner/Python37
    /cygdrive/c/Users/wturner/Python37/python.exe Scripts/pywin32_postinstall.py -install

dos:
    C:\Users\wturner\Python37>python.exe -i Scripts\pywin32_postinstall.py -install

    Scripts\pywin32_postinstall.py:156: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

      import imp
    Copied pythoncom37.dll to C:\Windows\system32\pythoncom37.dll
    Copied pywintypes37.dll to C:\Windows\system32\pywintypes37.dll
    Registered: Python.Interpreter
    Registered: Python.Dictionary
    Registered: Python
    -> Software\Python\PythonCore\3.7\Help[None]=None
    -> Software\Python\PythonCore\3.7\Help\Pythonwin Reference[None]='C:\\Users\\wturner\\Python37\\Lib\\site-packages\\PyWin32.chm'
    Pythonwin has been registered in context menu
    Shortcut for Pythonwin created
    Shortcut to documentation created
    The pywin32 extensions were successfully installed.
    >>>

Hmm..., ouput seems same as that from previous install

copied the example from https://github.com/cherrypy/tools/blob/master/WindowsService as MyService.py

    WTurner@ASDI-DB ~/swim2018/service
    $ /cygdrive/c/Users/wturner/Python37/python.exe MyService.py

    Usage: 'MyService.py [options] install|update|remove|start [...]|stop|restart [...]|debug [...]'
    Options for 'install' and 'update' commands only:
     --username domain\username : The Username the service is to run under
     --password password : The password for the username
     --startup [manual|auto|disabled|delayed] : How the service starts, default = manual
     --interactive : Allow the service to interact with the desktop.
     --perfmonini file: .ini file to use for registering performance monitor data
     --perfmondll file: .dll file to use when querying the service for
       performance data, default = perfmondata.dll
    Options for 'start' and 'stop' commands only:
     --wait seconds: Wait for the service to actually start or stop.
                     If you specify --wait with the 'stop' option, the service
                     and all dependent services will be stopped, each waiting
                     the specified period.

cherrypy as a service

have edited files, so here goes...

as a test:

    WTurner@ASDI-DB ~/swim2018/service

    $ /cygdrive/c/Users/wturner/Python37/python.exe SwimService.py

    Usage: 'SwimService.py [options] install|update|remove|start [...]|stop|restart [...]|debug [...]'

    Options for 'install' and 'update' commands only:
     --username domain\username : The Username the service is to run under
     --password password : The password for the username
     --startup [manual|auto|disabled|delayed] : How the service starts, default = manual
     --interactive : Allow the service to interact with the desktop.
     --perfmonini file: .ini file to use for registering performance monitor data
     --perfmondll file: .dll file to use when querying the service for
       performance data, default = perfmondata.dll
    Options for 'start' and 'stop' commands only:
     --wait seconds: Wait for the service to actually start or stop.
                     If you specify --wait with the 'stop' option, the service
                     and all dependent services will be stopped, each waiting
                     the specified period.

(note: SwimService.py and srv.conf are in the same directory)

here goes the install:

    WTurner@ASDI-DB ~/swim2018/service

    $ /cygdrive/c/Users/wturner/Python37/python.exe SwimService.py install

    Installing service CherryPyService
    Service installed

and to view:

-> Control Panel -> Administrative Tools -> Services
and there is a "CherryPy Service" there!!!

Yipee!

it seems to have created (empty) log files:

WTurner@ASDI-DB /cygdrive/c/temp
$ lr
...
-rwxr-x---+ 1 WTurner         Domain Users          0 Feb 25 22:17 cherrypy_error.log
-rwxr-x---+ 1 WTurner         Domain Users          0 Feb 25 22:17 cherrypy_access.log

To start, it seems that I can do either of:

from command line:
- $ /cygdrive/c/Users/wturner/Python37/python.exe SwimService.py start
from the 'Services (Local)' window:
- -> Cherrypy Service -> Start

Here goes:

-> Cherrypy Service -> Start

Windows could not start the CherryPy Service service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.

and the same for command line:

    WTurner@ASDI-DB ~/swim2018/service
    $ /cygdrive/c/Users/wturner/Python37/python.exe SwimService.py start
    Starting service CherryPyService
    Error starting service: The service did not respond to the start or control request in a timely fashion.

wtf?

search yields this:

add to SYSTEM PATH (not USER PATH):
- C:\Users\wturner\Python37
- C:\Users\wturner\Python37\Lib\site-packages\win32
- C:\Users\wturner\Python37\Lib\site-packages\pywin32_system32

http://www.geekswithblogs.net/renso/archive/2009/10/21/how-to-set-the-windows-path-in-windows-7.aspx

Answer:

The path is now managed by Windows 7 and not the autoexec.bat or
autoexec.nt files. To change the system environment variables, follow
the below steps.

    From the desktop, right-click My Computer and click Properties.

    In the System Properties window, click on the Advanced tab.

    In the Advanced section, click the Environment Variables button.

    Finally, in the Environment Variables window (as shown below),
highlight the Path variable in the Systems Variable section and
click the Edit button. Add or modify the path lines with the paths
you wish the computer to access. Each different directory is
separated with a semicolon as shown below.

-> right click on window -> System -> Advanced System Settings -> Advanced -> Environment Variables -> System Variables -> Path -> Edit

Current Path :

C:\Program Files (x86)\Common Files\Oracle\Java\javapath;
C:\ProgramData\Oracle\Java\javapath;
C:\app\plucic\product\12.1.0\dbhome_1\bin;
%SystemRoot%\system32;
%SystemRoot%;
%SystemRoot%\System32\Wbem;
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;
C:\Program Files\Dell\SysMgt\oma\bin;
C:\Program Files\Dell\SysMgt\shared\bin;
C:\Program Files\Dell\SysMgt\idrac;
C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;
C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;
C:\Program Files\Git\cmd;
C:\Program Files\Maven\apache-maven-3.3.3\bin\;
C:\ProgramData\chocolatey\bin;
C:\Gradle\gradle-4.9\bin;
C:\Program Files\nodejs\

added

    C:\Users\wturner\Python37

and I think I added these, but am not sure if they were needed:

    C:\Users\wturner\Python37\Lib\site-packages\win32
    C:\Users\wturner\Python37\Lib\site-packages\pywin32_system32

and the service starts, but only gets this far:

$ cat cherrypy_error.log
[26/Feb/2019:08:34:38] ENGINE Bus STARTING
[26/Feb/2019:08:34:38] ENGINE Serving on http://0.0.0.0:8080
[26/Feb/2019:08:34:38] ENGINE Bus STARTED

HOWEVER, this works:

    http://asdi-db.cssiinc.com:8080/hello/
    Hello world!

and it seems I had the config and code goofed up; they have been edited...

and edited SwimService.py for tools.staticdir items and also srv.conf

and it works! YIPEE !!!: http://asdi-db.cssiinc.com:8080/track.html

and favicon works too.

service executable is: C:\Users\wturner\Python37\lib\site-packages\win32\PythonService.exe which means that the runtime working directory is C:\Users\wturner\Python37\lib\site-packages\win32\

and committed to git:

git commit <filename> # to update a file in the local branch
git add <filename> # to add a new file to the local branch (still needs commit)
git push # to push local branch to master github repo
```
wendell-t
zaq12wsx
```

finished.

logging

2/28/2019: have changed over use Python's logging system.

12/11/2019: use TPA

1) vim CssiConsumer.java

            String asdex_filter = "KTPA";    // todo: move this elsewhere...
            String q02_filter = "TPA";    // todo: move this elsewhere...

2) kill current processes

    WTurner@ASDI-DB ~/swim2018
    $ /cygdrive/c/Users/wturner/Python37/python.exe  manage.py

    now: 2019-12-12 03:28 UTC
    w == start a swim process
    Enter input: l
    e-s psutil.Process(pid=9896, name='java.exe', started='09:00:12')
    e-a psutil.Process(pid=4972, name='java.exe', started='09:00:12')
    e-f psutil.Process(pid=9036, name='java.exe', started='09:00:12')
    e-r psutil.Process(pid=11700, name='java.exe', started='09:00:12')

    now: 2019-12-12 03:28 UTC
    Enter input: k 9896
    exception!

    now: 2019-12-12 03:28 UTC
    Enter input: k 4972
    exception!

    now: 2019-12-12 03:28 UTC
    Enter input: k 9036
    exception!

    now: 2019-12-12 03:28 UTC
    Enter input: k 11700
    exception!

    now: 2019-12-12 03:29 UTC
    w == start a swim process

exception! didn't seem to matter...

3a) start q02 aka stdds

    WTurner@ASDI-DB ~/swim2018/endpoint/bin
    $ ./run_swim_listener.sh  stdds
    + echo 'doing stdds'
    doing stdds

looks ok...

3b) start q03 aka asdex

hmm... filtering on KTPA caused errors, so TURNED FILTER OFF

HELP:

WTurner@ASDI-DB ~/swim2018/endpoint/cron
$ psql -h localhost -p 5433 swim postgres
Password for user postgres:
psql (10.5)
Type "help" for help.

to search asdex:

swim=# select distinct(src) from asdex_20191212 where time > '2019-12-12T04:06:00 UTC' order by src;
 src
------
 KATL
 KBOS
 KBWI
 KCLE
 KCLT
 KCVG
 KDCA
 KDEN
 KDFW
 KDTW
 KEWR
 KFLL
 KHOU
 KIAD
 KIAH
 KJFK
 KLAS
 KLAX
 KLGA
 KMCI
 KMCO
 KMDW
 KMEM
 KMIA
 KMKE
 KMSP
 KMSY
 KORD
 KPDX
 KPHL
 KPHX
 KPIT
 KPVD
 KSAN
 KSDF
 KSEA
 KSFO
 KSLC
 KSNA
 KSTL
 PHNL
(41 rows)

to search stdds:

swim=# select distinct(src_airport) from stdds_20191212 where mrttime > '2019-12-12T03:06:00 UTC' order by src_airport;
 src_airport
-------------
 A11
 A80
 A90
 ABE
 ABI
 ABQ
 ACT
 ACY
 AGS
 ALB
 ALO
 AMA
 ASE
 AUS
 AVL
 AVP
 AZA
 AZO
 BFL
 BGM
 BGR
 BHM
 BIL
 BIS
 BNA
 BOI
 BTR
 BTV
 BUF
 C90
 CAE
 CHA
 CHS
 CID
 CKB
 CLE
 CLT
 CMH
 CMI
 COS
 CPR
 CRP
 CRW
 CVG
 D01
 D10
 D21
 DAB
 DLH
 DSM
 ELM
 ELP
 EUG
 EVV
 F11
 FAI
 FAR
 FAT
 FAY
 FLO
 FSD
 FSM
 FWA
 GEG
 GGG
 GPT
 GRB
 GSO
 GSP
 GTF
 HSV
 HTS
 HUF
 I90
 ICT
 ILM
 IND
 JAN
 JAX
 L30
 LBB
 LCH
 LEX
 LFT
 LIT
 M03
 M98
 MAF
 MCI
 MDT
 MGM
 MIA
 MKE
 MLI
 MLU
 MOB
 MSN
 MSO
 MSY
 MWH
 MYR
 N90
 NCT
 OKC
 ORF
 P31
 P50
 P80
 PBI
 PCT
 PHL
 PIA
 PIT
 PSC
 PVD
 PWM
 R90
 RDG
 RDU
 RFD
 ROA
 ROC
 ROW
 RST
 RSW
 S46
 S56
 SAT
 SAV
 SBA
 SBN
 SCT
 SDF
 SGF
 SHV
 SPI
 SUX
 SYR
 T75
 TLH
 TOL
 TPA
 TRI
 TUL
 TYS
 U90
 Y90
 YNG
(148 rows)

WAIT, both F11 and TPA???