Nov 22
get ciws create statements on asdi-db:
pg_dump --schema-only --table=ciws_obs --create --no-owner --dbname ciwsdb
pg_dump --schema-only --table=ciws_fcst --create --no-owner --dbname ciwsdb
which was close, but needed editing to be:
CREATE TABLE ciws_obs (
ctime timestamp with time zone,
id integer,
wtype text,
wx_geog geography(Polygon,4326)
);
CREATE TABLE ciws_fcst (
fcst_id text,
wtype text,
analysis_time timestamp with time zone,
sampling_time timestamp with time zone,
wx_geog geography(MultiPolygon,4326)
);
Note: all tables on cev are in same database: "locs"
it looks like everything will have to be done by the (one and only) 'locs' user:
locs=> create user ciwsuser with password 'xxxxxxx';
ERROR: permission denied to create role
will need to create an SSH Tunnel as described here
on asdi-db in terminal window: (note: local port is 55432)
ssh wendell@opal4.opalstack.com -L 55432:127.0.0.1:5432 -N
on asdi-db in client program, access d.b. with these environment credentials:
CEV_HOST=127.0.0.1
CEV_PORT=55432
CEV_DATABASE=locs
CEV_USER=locs
CEV_PASSWORD=password
and then to view:
locs=> select ctime, id, wtype, substr(st_astext(wx_geog),1,40) from ciws_obs limit 10;
ctime | id | wtype | substr
------------------------+----+-------+------------------------------------------
2020-11-23 00:30:00+00 | 2 | vil | POLYGON((-81.7312336333774 23.7883133564
2020-11-23 00:30:00+00 | 9 | vil | POLYGON((-98.633847506355 25.49075757489
2020-11-23 00:30:00+00 | 10 | vil | POLYGON((-81.1840144480391 24.2503790311
2020-11-23 00:30:00+00 | 12 | vil | POLYGON((-80.8664158394034 24.3643736691
2020-11-23 00:30:00+00 | 13 | vil | POLYGON((-80.3393670222702 24.2798963838
2020-11-23 00:30:00+00 | 15 | vil | POLYGON((-79.9210167737945 24.2655656096
2020-11-23 00:30:00+00 | 22 | vil | POLYGON((-79.8182081387151 24.5212847937
2020-11-23 00:30:00+00 | 27 | vil | POLYGON((-98.7573005653902 26.1411698451
2020-11-23 00:30:00+00 | 28 | vil | POLYGON((-80.5466446055401 24.8589836467
2020-11-23 00:30:00+00 | 30 | vil | POLYGON((-81.1352544065028 24.9518311076
(10 rows)
Yipee!! CIWS is inserted into d.b. on cev!
WAIT!: the individual CELLS (from .csv?) are the ones needed for LEGOs!
COPY over files from ]reproject ol example](https://openlayers.org/en/latest/examples/reprojection.html)
setup site to alter between 'lego_vue' and 'lego_examp'
wendell@opal4:~/src/lego_example
> npm install
> npm run-script build
> cp dist/* ~/apps/angela/lego_examp/
PROBLEM: 'lego_examp' subdir is not used :-(
need to have (stupid) csv data in d.b. with row/col
for now: VIL only, no ET.
CREATE TABLE csv_vil (
gtime timestamptz,
row smallint,
col smallint,
maxval smallint,
leadzero smallint,
value text );
CREATE TABLE
IS proceeding fine on asdi-db...
and to try to select all polys...
ciwsdb=> select * from csv_vil where gtime = '2020-11-05 23:10:00+00' and maxval >= 4;
gtime | row | col | maxval | leadzero | value
------------------------+-----+-----+--------+----------+--------------------------------------
2020-11-05 18:10:00-05 | 69 | 718 | 4 | 0 | 222322333332344332344222333222333322
2020-11-05 18:10:00-05 | 75 | 682 | 4 | 0 | 112222122233222344222234222233122233
2020-11-05 18:10:00-05 | 87 | 713 | 4 | 0 | 222222333322333443223343222333223333
2020-11-05 18:10:00-05 | 89 | 670 | 4 | 0 | 221111122211112221113333122333122243
this seems to work well:
SELECT G.row_id, G.column_id, st_astext(G.poly) FROM ciws_grid G, csv_vil V
WHERE V.row = G.row_id and V.col = G.column_id
AND V.gtime = '2020-11-05 23:10:00+00' and V.maxval >= 5;
TODO: Q: how to use geojson as "transfer mechanism" and still use poly corner coords in Image calls in ol??? Q: how to auto insert poll into both? * Q: how to copy all to cev?
on asdi-db:
ciwsdb=> \copy csv_vil to 'trans_to_cev.csv' csv header;
COPY 3237297
$ gzip trans_to_cev.csv
$ scp trans_to_cev.csv.gz wendell@carrotseverywhere.com:cssi/
on cev (with approx. create statements):
gzip -d trans_to_cev.sql.gz
> sed 's/ciwsdb/locs/g' trans_to_cev.sql > load_vil.sql
> gzip -d trans_to_cev.csv.gz
locs=> delete from csv_vil ;
DELETE 531415
locs=> \copy csv_vil from 'trans_to_cev.csv' csv header;
COPY 3237297
note: postg.py replaces '_' with ' '
https://lego.carrotseverywhere.com/get_ciws_csv?gtime=2020-11-12_18:10:00-0000&wtype=vil&thresh=5
seems fine... continuing...
ON HOLD: don't need vue.js, JUST plain npm + parcel (v2)
new instructions from Resolving EACCES permissions errors
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
export PATH=~/.npm-global/bin:$PATH
-- or --
export NPM_CONFIG_PREFIX=~/.npm-global
Now, will start over:
mkdir legos
cd legos/
npm init
<std answers>
and: (note: '@next' == bleeding edge repo)
npm install -g @vue/cli
npm install -g @vue/cli-service
npm install vuelayers@next
and COPY over working items from current vue app:
cp ~/src/from_wf/locator/vue.config.js .
cp ~/src/from_wf/locator/.env .
cd src
cp ~/src/from_wf/locator/src/main.js .
cp ~/src/from_wf/locator/src/App.vue .
cd components
cp ~/src/from_wf/locator/src/components/AppOL.vue .
> npm install
npm WARN vuelayers@0.12.0-rc.14 requires a peer of vue@^2.3 but none is installed. You must install peer dependencies yourself.
npm WARN vuequery@2.1.1 requires a peer of vue@~2.5.0 but none is installed. You must install peer dependencies yourself.
npm WARN legos@1.0.0 No repository field.
???
Fundamental design principals
All great visuals follow three key principles: less is more, attract attention, and have impact. In other words, any feature or design you include in your plot to make it more attractive or pleasing should support the message that the plot is meant to get across and not distract from it.
The National CIWS gridded product files are mapped using the Lambert Azimuthal Equal Area projection.
in document text:
int grid_mapping0 ;
grid_mapping0:grid_mapping_name = “lambert_azimuthal_equal_area” ;
grid_mapping0:long_name = “Lambert Azimuthal Equal Area Projection” ;
grid_mapping0:latitude_of_projection_origin = 38. ;
grid_mapping0:longitude_of_projection_origin = -98. ;
grid_mapping0:false_easting = 0. ;
grid_mapping0:false_northing = 0. ;
grid_mapping0:earth_radius = 6370997. ;
in C++ code:
int numRows = 3520;
int numCols = 5120;
double eastSpacing_m = 1000.;
double northSpacing_m = 1000.;
double centerLat_deg = 38.0;
double centerLon_deg = -98.0;
double falseEasting_m = 0.0;
double falseNorthing_m = 0.0;
double earthRadiusSpereical_m = 6370997.; /* see sphdz.c */
The EPSG code for LAEA is 3035. Further details available here: https://spatialreference.org/ref/epsg/etrs89-etrs-laea/
their example proj4 string is:
+proj=laea +lat_0=52 +lon_0=10 +x_0=4321000 +y_0=3210000 +ellps=GRS80 +units=m +no_defs
FIRST POSSIBLE lead on answer: https://gis.stackexchange.com/questions/121960/mapping-netcdf-cf-mercator-parameters-to-gdal
"Given the ambiguity, in the end I did this directly in proj4:"
import pyproj
p=pyproj.Proj(r'+proj=merc +lat_ts=30.0 +lon_0=-128.735 +a=6370997.0 +b=6370997.0')
print p(x_merc,y_merc,inverse=True)
SECOND possible: https://gis.stackexchange.com/questions/113980/custom-projection-problem-with-proj-4
+proj=laea +lat_0=38 +lon_0=-98 +x_0=0 +y_0=0 +a=6370997 +b=6370997 +units=m +no_defs
pip3.7 install Cartopy
Collecting Cartopy
Downloading Cartopy-0.18.0.tar.gz (14.4 MB)
|xxxxxxxxxxxxxxxxxxxx| 14.4 MB 841 kB/s
Requirement already satisfied: numpy>=1.10 in /usr/lib/python3.7/site-packages (from Cartopy) (1.19.0)
Requirement already satisfied: shapely>=1.5.6 in /usr/lib/python3.7/site-packages (from Cartopy) (1.6.4.post2)
Requirement already satisfied: pyshp>=1.1.4 in /usr/lib/python3.7/site-packages (from Cartopy) (2.1.0)
Requirement already satisfied: six>=1.3.0 in /usr/lib/python3.7/site-packages (from Cartopy) (1.15.0)
Requirement already satisfied: setuptools>=0.7.2 in /usr/lib/python3.7/site-packages (from Cartopy) (46.4.0)
Using legacy 'setup.py install' for Cartopy, since package 'wheel' is not installed.
Installing collected packages: Cartopy
Running setup.py install for Cartopy ... done
Successfully installed Cartopy-0.18.0
That was too easy...
notebooks from here
retrieve
cd cartopy_testing/
git clone https://github.com/geohackweek/tutorial_contents.git
cd ~/gis/brick/cartopy_testing/tutorial_contents/visualization/notebooks
> jupyter notebook
http://localhost:8888/tree?token=d89c8446b70fb1a90a5f330ce83011b8b390d6e3dcdee27d
ModuleNotFoundError: No module named 'pykdtree'
> pip3.7 install pykdtree
ModuleNotFoundError: No module named 'scipy'
> pip3.7 install scipy
(is taking a LONG time)
^C out of it
from here
ModuleNotFoundError: No module named 'scipy'
ok, scipy:
> pip3.7 install scipy
Collecting scipy
Using cached scipy-1.5.4.tar.gz (25.2 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python3.7 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp6ifero8i
cwd: /tmp/pip-install-9wmzqplq/scipy
libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
NOT AVAILABLE
ERROR: Command errored out with exit status 1: /usr/bin/python3.7 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp6ifero8i Check the logs for full command output.
which requires:
python -m pip install --user numpy scipy matplotlib ipython jupyter pandas sympy nose
remaining:scipy sympy nose
> pip3.7 install numpy
Requirement already satisfied: numpy in /usr/lib/python3.7/site-packages (1.19.0)
> pip3.7 install matplotlib
Requirement already satisfied: matplotlib in /usr/lib/python3.7/site-packages (3.3.2)
> pip3.7 install pandas
Requirement already satisfied: pandas in /usr/lib/python3.7/site-packages (0.24.2)
> pip3.7 install jupyter
Requirement already satisfied: jupyter in /usr/lib/python3.7/site-packages (1.0.0)
POSSIBLE location of wheel / binary Unofficial Windows Binaries for Python Extension Packages
Nope: that is a Windows wheel, incompat. w/Cygwin:
> pip3.7 install scipy-1.5.4-cp37-cp37m-win_amd64.whl
ERROR: scipy-1.5.4-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.
https://geopandas.readthedocs.io/en/latest/gallery/cartopy_convert.html
RSError: Invalid projection: +init=epsg:4326 +type=crs: (Internal Proj Error:
proj_create: cannot build geodeticCRS 4326: SQLite error on SELECT name,
ellipsoid_auth_name, ellipsoid_code, prime_meridian_auth_name,
prime_meridian_code, area_of_use_auth_name, area_of_use_code, publication_date,
deprecated FROM geodetic_datum WHERE auth_name = ? AND code = ?: no such
column: publication_date)
I got that same error when using Proj-5.x. It seems that the 'publication_date' column is a Proj-6 or Proj-7 item (which both require SQLite.)
TODO: read this: GeoPandas now uses pyproj.CRS and catches up with PROJ 6
where Joris Van den Bossche says:
With the changes in PROJ 6, the PROJ community wants to move away from using proj4 strings to represent a CRS, as we did up to now. Why? Because the proj4 string is limited and cannot faithfully describe a certain CRS. Using a proj4 string (instead of e.g. a WKT string) loses valuable information about the CRS (e.g. the name, the exact datum, the area of use, etc), possibly resulting in less precise transformations.
>>> import pyproj
>>> pyproj.__version__
'2.1.3'
Q: Cygwin says proj (and libproj-devel) 6.3.1-1 is installed, but above error seems to indicate proj < 6 is being used :-(
so, pyproj must have been installed via pip
so:
geopandas 0.7.0 GOES WITH Proj 6
geopandas 0.6.x GOES WITH Proj <6
we're using:
>>> import geopandas
>>> geopandas.__version__
'0.6.3'
SO, it should be ok to upgrade to geopandas 0.7, right???
WILL try to upgrade pyproj:
pip3.7 install pyproj==2.2.0 Collecting pyproj==2.2.0 Downloading pyproj-2.2.0.tar.gz (378 kB) ERROR: Minimum supported proj version is 6.1.0, installed version is 6.0.0.
Cygwin:
libproj13: Uninstall 5.2.0-1 libproj15: Reinstall 6.3.1-1 libproj-devel: Reinstall 6.3.1-1 proj: Reinstall 6.3.1-1
Problem 1/1 package libgeotiff2-1.4.3-1 requires libproj13, but none of the providers can be installed Solution 1/2 - allow replacement of libgeotiff2-1.4.3-1 with libgeotiff2-1.4.2-2 - allow deinstallation of libgdal20-2.4.0-1 Solution 2/2 (default) - do not ask to deinstall libproj13-5.2.0-1
compatibility for pyproj and Proj is here
pyproj 3.0.0 github says: Minimum supported PROJ version is 7.2.0
> cat /usr/local/lib/pkgconfig/proj.pc
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
Name: proj
Description: Cartographic Projections Library.
Requires:
Version: 6.0.0
Libs: -L${libdir} -lproj
Cflags: -I${includedir}
WHICH WAS INSTALLED FROM SOURCES!!!!!!!!
at 7:39pm:
get proj-6.3.2.tar.gz
./configure
make
took about an hour
make install
> cat /usr/lib/pkgconfig/proj.pc
prefix=/usr
exec_prefix=/usr
libdir=${exec_prefix}/lib
includedir=${prefix}/include
datadir=${prefix}/share/proj
Name: proj
Description: Cartographic Projections Library.
Requires:
Version: 6.3.1
Libs: -L${libdir} -lproj
Libs.Private: -lsqlite3 -lstdc++
Cflags: -I${includedir}
and back to try to install a pyproj that will use it:
> pip3.7 install pyproj==2.2.0
Collecting pyproj==2.2.0
Successfully uninstalled pyproj-2.1.3
Successfully installed pyproj-2.2.0
Yipee: !!!
[1]
# sphinx_gallery_thumbnail_number = 7
import matplotlib.pyplot as plt
import geopandas
from cartopy import crs as ccrs
path = geopandas.datasets.get_path('naturalearth_lowres')
df = geopandas.read_file(path)
# Add a column we'll use later
df['gdp_pp'] = df['gdp_md_est'] / df['pop_est']
[2]
df.plot()
[3]
# Define the CartoPy CRS object.
crs = ccrs.AzimuthalEquidistant()
# This can be converted into a `proj4` string/dict compatible with GeoPandas
crs_proj4 = crs.proj4_init
df_ae = df.to_crs(crs_proj4)
[4]
# Here's what the plot looks like in GeoPandas
df_ae.plot()
however, this one barfs because of scipy:
[5]
fig, ax = plt.subplots(subplot_kw={'projection': crs})
ax.add_geometries(df_ae['geometry'], crs=crs)
this barfs because: ModuleNotFoundError: No module named 'pyepsg'
crs_epsg = ccrs.epsg('3857')
df_epsg = df.to_crs(epsg='3857')
# Generate a figure with two axes, one for CartoPy, one for GeoPandas
fig, axs = plt.subplots(1, 2, subplot_kw={'projection': crs_epsg},
figsize=(10, 5))
# Make the CartoPy plot
axs[0].add_geometries(df_epsg['geometry'], crs=crs_epsg,
facecolor='white', edgecolor='black')
# Make the GeoPandas plot
df_epsg.plot(ax=axs[1], color='white', edgecolor='black')
DON'T remember if I tried this: Vitosh Academy Python – Making Maps with Cartopy
Let's try building scipy from source just for fun:
git clone https://github.com/scipy/scipy.git scipy
> python setup.py build
Error: 'pybind11' must be installed before running the build.
> pip3.7 install pybind
Collecting pybind
Traceback (most recent call last):
AttributeError: 'ParsedRequirement' object has no attribute 'req'
Back to scipy from pip, maybe need Cygwin's lapack:
liblapack0: installed
liblapack-devel: new
libopenblas: installed
an will try an ANCIENT version of scipy:
> pip3.7 install scipy==0.15.0
NOT AVAILABLE
libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
libraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
libraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
libraries flame not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
libraries lapack_atlas not found in /usr/local/lib
libraries tatlas,tatlas not found in /usr/local/lib
libraries lapack_atlas not found in /usr/lib
libraries tatlas,tatlas not found in /usr/lib
libraries lapack_atlas not found in /usr/lib/
libraries tatlas,tatlas not found in /usr/lib/
more suggestions: Installing SciPy, NumPy and matplotlib Under Cygwin
but nothing successful...
Seems like it would be pretty if it worked:
https://scitools.org.uk/cartopy/docs/v0.15/matplotlib/intro.html
ModuleNotFoundError: No module named 'scipy'
Tiling in projections other than spherical mercator (the projection used in Leaflet by default) is a complex topic, and standards are often lacking. To get a better grasp of the concepts involved, you might want to refer to The Hitchhacker’s Guide To Tiled Maps