Note: this page, Databases and data access APIs lists various ways of reading OSM data into databases. They list these among choices:
I think all of these are going to get used.
more installation notes:
-f Postgres, so make sure GDAL compilation has Posggresql: yes in itlibosmium is use extensivelyMost of the steps are from:
1) set up terminal session:
at the beginning of each terminal session, start a python venv:
cd ~/openmaptiles/vmpython
source bin/activate
export LD_LIBRARY_PATH=/home/wendell/usr/local/lib64:/home/wendell/usr/local/lib
2) ensure our own PosgGreSQL is running
ps aux
if not, then (not necessarily in a venv session):
cd ~/openmaptiles
./start_stop_pg.sh start
3) psql to our d.b.:
use our psql, not system one:
unset PGPORT
unset PGUSER
unset PGDATABASE
unset PGPASSWORD
unset PGHOST
psql --port 55432 openmaptiles
4) if re-compilation needed
use these for new packages:
./configure --prefix=/home/wendell/usr/local
cmake -DCMAKE_INSTALL_PREFIX=/home/wendell/usr/local/ ..
PKG_CONFIG_PATH=/home/wendell/usr/local/lib/pkgconfig:/home/wendell/usr/local/lib64/pkgconfig
text notes describing these steps are on my work laptop in:
~/gis/openmaptiles/opalstack/notes_opalstack.txt~/gis/openmaptiles/opalstack/notes_omt_libraries.txt~/gis/openmaptiles/opalstack/notes_postgis.txtmanually performed, script not used
1) ====> : Filtering border from PBF lanet-latest.osm.pbf to osmborder_lines.pbf
run a cpp program from osmborder (from git?)
from osmborder's readme.md:
OSMBorder extracts the admin boundary data from an OSM planet file and assembles all the pieces into linestrings for use in map renderers etc.
osmborder_filter -o /home/wendell/openmaptiles/wdb-map-gen/data/osmborder_lines.pbf /home/wendell/openmaptiles/wdb-map-gen/data/planet-latest.osm.pbf
then create a .csv file from that .pbf file
2) ====> : Creating CSV osmborder_lines.csv from osmborder_lines.pbf
osmborder -o /home/wendell/openmaptiles/wdb-map-gen/data/osmborder_lines.csv /home/wendell/openmaptiles/wdb-map-gen/data/osmborder_lines.pbf
seems to have succeeded:
du -sh data/*.csv
3.4G data/osmborder_lines.csv
possibly already done by previous run of the load_water step
cd ~/openmaptiles/wdb-map-gen/
note: opalstack is slow, it may be faster to d/l to rserver, then scp from there to opalstack
download 723 MB file from osmdata.openstreetmap.de
wget https://osmdata.openstreetmap.de/download/water-polygons-split-4326.zip -P data
unzip -o data/water-polygons-split-4326.zip -d data
this produces a directory of one hugh shapefile:
> file water-polygons-split-4326/*
water-polygons-split-4326/README.txt: ASCII text
water-polygons-split-4326/water_polygons.cpg: ASCII text
water-polygons-split-4326/water_polygons.dbf: DBase 3 data file (53271 records)
water-polygons-split-4326/water_polygons.prj: ASCII text, with no line terminators
water-polygons-split-4326/water_polygons.shp: ESRI Shapefile version 1000 length 542279024 type Polygon
water-polygons-split-4326/water_polygons.shx: ESRI Shapefile version 1000 length 213134 type Polygon
load shapefiles into postgis
using ogr2ogr (compiled with 'Postgresql:yes'), load those shapefiles into postgis:
ogr2ogr -progress -f Postgresql \
-s_srs EPSG:3857 -t_srs EPSG:3857 \
-lco OVERWRITE=YES -lco GEOMETRY_NAME=geometry -overwrite \
-nln osm_ocean_polygon -nlt geometry \
--config PG_USE_COPY YES \
'PG:dbname=openmaptiles user=tiles host=localhost password=xxxx port=55432' \
data/water-polygons-split-4326/water_polygons.shp
0...10...20...30...40...50...60...70...80...90...100 - done.
which creates this PostGIS table:
> psql -p 55432 openmaptiles
openmaptiles=# \d osm_ocean_polygon
Table "public.osm_ocean_polygon"
Column | Type | Collation | Nullable | ...
----------+-------------------------+-----------+----------+-
ogc_fid | integer | | not null | ...
x | numeric(9,0) | | |
y | numeric(9,0) | | |
geometry | geometry(Geometry,3857) | | |
Indexes:
"osm_ocean_polygon_pkey" PRIMARY KEY, btree (ogc_fid)
"osm_ocean_polygon_geometry_geom_idx" gist (geometry)
openmaptiles=# select count(*) from osm_ocean_polygon;
53271
things around eastern seaboard:
openmaptiles=# select ogc_fid, x, y
from osm_ocean_polygon
where x < -76 and x > -80 and y > 36 and y < 38;
ogc_fid | x | y
---------+-----+----
7251 | -78 | 37
7252 | -78 | 37
7253 | -78 | 37
7254 | -78 | 37
7255 | -78 | 37
8865 | -77 | 37
...
picking one at random:
openmaptiles=# select ST_AsGeoJson(geometry) from osm_ocean_polygon where ogc_fid = 8867;
{"type":"Polygon","coordinates":[[[-76.533374,38.0002866],[...
which is a small portion of the Chesapeake Bay.
Note: geom is 3857, don't know if that is going to matter later.
seems good so far.
import data from osm world file (we're using the dallas-area crop until it works)
apparently this impostm import does this:
tools/imposm3/bin/imposm import \
-connection postgis://tiles:xxxx@localhost:55432/openmaptiles \
-mapping /home/wendell/openmaptiles/wdb-map-gen/config/lake_centerlines.yaml \
-overwritecache \
-cachedir /home/wendell/openmaptiles/wdb-map-gen/data/imposm3_cache \
-read /home/wendell/openmaptiles/wdb-map-gen/data/dallas.osm.pbf \
-dbschema-import=public \
-write
[2021-10-02T15:28:57Z] 0:00:00 [info] removing existing cache /home/wendell/openmaptiles/wdb-map-gen/data/imposm3_cache
[2021-10-02T15:28:57Z] 0:00:00 [step] Starting: Imposm
[2021-10-02T15:28:57Z] 0:00:00 [step] Starting: Reading OSM data
[2021-10-02T15:28:57Z] 0:00:00 [info] reading /home/wendell/openmaptiles/wdb-map-gen/data/dallas.osm.pbf with data till 2021-09-05 23:59:59 +0000 UTC
[2021-10-02T15:29:14Z] 0:00:16 [progress] 16s C: 0/s (23036916) N: 0/s (4) W: 0/s (3371824) R: 0/s (5366)
[2021-10-02T15:29:14Z] 0:00:17 [step] Finished: Reading OSM data in 16.884572337s
[2021-10-02T15:29:14Z] 0:00:17 [step] Starting: Importing OSM data
[2021-10-02T15:29:14Z] 0:00:17 [step] Starting: Writing OSM data
[2021-10-02T15:29:25Z] 0:00:27 [step] Finished: Writing OSM data in 10.744546117s
[2021-10-02T15:29:25Z] 0:00:27 [step] Starting: Creating generalized tables
[2021-10-02T15:29:25Z] 0:00:27 [progress] 10s C: 0/s ( 0.0%) N: 0/s (100.0%) W: 0/s (100.0%) R: 0/s (100.0%)
[2021-10-02T15:29:25Z] 0:00:27 [step] Finished: Creating generalized tables in 322.01µs
[2021-10-02T15:29:25Z] 0:00:27 [step] Starting: Creating geometry indices
[2021-10-02T15:29:25Z] 0:00:27 [step] Starting: Creating geometry index on osm_lake_polygon
[2021-10-02T15:29:25Z] 0:00:27 [step] Finished: Creating geometry index on osm_lake_polygon in 2.375729ms
[2021-10-02T15:29:25Z] 0:00:27 [step] Finished: Creating geometry indices in 2.624851ms
[2021-10-02T15:29:25Z] 0:00:27 [step] Finished: Importing OSM data in 10.747612098s
[2021-10-02T15:29:25Z] 0:00:27 [step] Finished: Imposm in 27.632324887s
the mapping yaml file is:
> cat config/lake_centerlines.yaml
tables:
lake_polygon:
fields:
- name: osm_id
type: id
- name: geometry
type: validated_geometry
- name: area
type: area
- name: name
key: name
type: string
mapping:
water:
- lake
type: polygon
and the PostGIS table is:
openmaptiles=# \d osm_lake_polygon
Table "public.osm_lake_polygon"
Column | Type | Collation | Nullable |
----------+-------------------------+-----------+----------+
id | integer | | not null |
osm_id | bigint | | not null |
area | real | | |
name | character varying | | |
geometry | geometry(Geometry,3857) | | |
Indexes:
"osm_lake_polygon_pkey" PRIMARY KEY, btree (osm_id, id)
"osm_lake_polygon_geom" gist (geometry)
and:
openmaptiles=# select id, osm_id, area, name from osm_lake_polygon limit 22;
id | osm_id | area | name
----+-----------+---------------+----------------------------------------
1 | -253990 | 6.6655e+07 | Lake Bridgeport
2 | -2136481 | 794332.9 | Echo Lake
3 | -2136483 | 2.1545932e+06 | Caddo Creek Lake
4 | -2142482 | 842893.1 | Old Lake
5 | -2136480 | 1.4044872e+06 | Rhine Lake
6 | -2142480 | 1.1932928e+06 | Terra Verda Lake
7 | -2142500 | 3.6061308e+06 | Trinidad Lake
8 | -2142485 | 3.391756e+06 | Koon Kreek Lake
9 | -2142285 | 2.610082e+06 | Soil Conservation Service Site 42 Lake
10 | -952396 | 1.1748455e+08 | Lake Ray Hubbard
11 | -2463236 | 6.147459e+06 | Lake Crook
12 | -2142396 | 1.8643422e+07 | Bardwell Lake
13 | -2578321 | 1.1506677e+08 | Lavon Lake
14 | -2136180 | 8.3540145e+06 | Lake Athens
15 | -2140470 | 2.097937e+07 | Benbrook Lake
16 | -4049995 | 28289.719 | Heritage Lake
17 | -937920 | 1.0371684e+08 | Jim Chapman Lake
18 | -6521733 | 221178.89 |
19 | -6904538 | 69343.21 | Wilemon Lake
20 | -8905912 | 1.6584081e+06 | Lake Viridian
and one of them looks like:
openmaptiles=# select ST_AsGeoJson(geometry) from osm_lake_polygon where name='Lake Bridgeport';
{"type":"Polygon","coordinates":[[[-10904350.3505353,3928761.52079316],
[-10904344.6214904,3928779.6629242],
[-10904332.1930088,3928788.26538367],
[-10904315.9576048,3928793.04081139],
[-10904296.839017,3928799.73533953],
HELP!!! What coordinates are these???
not sure where this geojson file came from...
not sure what we're doing here...
ogr2ogr -progress -f Postgresql \
-t_srs EPSG:3857 \
'PG:dbname=openmaptiles user=tiles host=localhost password=xxxx port=55432' \
-lco OVERWRITE=YES -overwrite \
-nln lake_centerline data/lake_centerline.geojson
0...10...20...30...40...50...60...70...80...90...100 - done.
which created this table:
openmaptiles=# \d lake_centerline
Table "public.lake_centerline"
Column | Type | Collation | Nullable |
--------------+---------------------------+-----------+----------+
ogc_fid | integer | | not null |
osm_id | double precision | | |
wkb_geometry | geometry(LineString,3857) | | |
Indexes:
"lake_centerline_pkey" PRIMARY KEY, btree (ogc_fid)
"lake_centerline_wkb_geometry_geom_idx" gist (wkb_geometry)
openmaptiles=# select ogc_fid, osm_id from lake_centerline limit 22;
ogc_fid | osm_id
---------+-----------
1 | -1520541
2 | -1082659
3 | -7213401
4 | -338774
34 | -1603199
26023 | 366984344
5 | -1125603
6 | -2194833
...
AGAIN, unknown coordinates:
openmaptiles=# select ST_AsGeoJson(wkb_geometry) from lake_centerline where ogc_fid = 5391;
{"type":"LineString","coordinates":[[16321515.0629178,10161155.438388],
[16321494.3388573,10161161.5248103],...
====> : Importing shape files into Postgresql
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
<lots of these>
====> : Start importing /home/wendell/openmaptiles/wdb-map-gen/data/natural_earth_vector.sqlite into Postgresql
(re-compiled GDAL yet again, and it finally worked):
ogr2ogr -progress -f Postgresql \
-s_srs EPSG:4326 -t_srs EPSG:3857 \
-clipsrc -180.1 -85.0511 180.1 85.0511
'PG:dbname=openmaptiles user=tiles host=.. password=.. port=55432' \
-lco GEOMETRY_NAME=geometry -lco OVERWRITE=YES -lco DIM=2 \
-nlt GEOMETRY -overwrite /h/w/o/w/data/natural_earth_vector.sqlite
0...10...20...30...40...50...60...70...80...90...100 - done.
Holy cow! 337 tables (and 337 sequences created):
List of relations
Schema | Name | Type | Owner
--------+------------------------------------------------+----------+---------
public | ne_10m_admin_0_antarctic_claim_limit_lines | table | tiles
public | ne_10m_admin_0_antarctic_claims | table | tiles
...
public | ne_10m_wgs84_bounding_box | table | tiles
public | ne_110m_admin_0_boundary_lines_land | table | tiles
...
public | ne_50m_admin_0_tiny_countries | table | tiles
public | ne_50m_admin_0_tiny_countries_scale_rank | table | tiles
...
public | ne_50m_airports | table | tiles
..
public | ne_50m_urban_areas | table | tiles
public | ne_50m_wgs84_bounding_box | table | tiles
and one of them looks like:
openmaptiles=# \d ne_50m_airports
Table "public.ne_50m_airports"
Column | Type | Collation | Nullable | Default
------------+-------------------------+-----------+----------+--------------------------------------------------
ogc_fid | integer | | not null | nextval('ne_50m_airports_ogc_fid_seq'::regclass)
scalerank | integer | | |
featurecla | character varying | | |
type | character varying | | |
name | character varying | | |
abbrev | character varying | | |
location | character varying | | |
gps_code | character varying | | |
iata_code | character varying | | |
wikipedia | character varying | | |
natlscale | double precision | | |
comments | character varying | | |
wikidataid | character varying | | |
name_en | character varying | | |
wdid_score | integer | | |
ne_id | bigint | | |
geometry | geometry(Geometry,3857) | | |
Indexes:
"ne_50m_airports_pkey" PRIMARY KEY, btree (ogc_fid)
"ne_50m_airports_geometry_geom_idx" gist (geometry)
and:
openmaptiles=# select type, name, abbrev, location, iata_code from ne_50m_airports where name like '%Dallas%';
type | name | abbrev | location | iata_code
-------+------------------------+--------+----------+-----------
major | Dallas-Ft. Worth Int'l | DFW | parking | DFW
major | Dallas-Ft. Worth Int'l | DFW | parking | DFW
major | Dallas-Ft. Worth Int'l | DFW | parking | DFW
major | Dallas-Ft. Worth Int'l | DFW | parking | DFW
major | Dallas-Ft. Worth Int'l | DFW | parking | DFW
major | Dallas-Ft. Worth Int'l | DFW | parking | DFW
(6 rows)
openmaptiles=# select type, name, abbrev, location, iata_code from ne_50m_airports where name like '%Chicago%';
type | name | abbrev | location | iata_code
-------+----------------------+--------+----------+-----------
major | Chicago O'Hare Int'l | ORD | terminal | ORD
major | Chicago O'Hare Int'l | ORD | terminal | ORD
major | Chicago O'Hare Int'l | ORD | terminal | ORD
major | Chicago O'Hare Int'l | ORD | terminal | ORD
major | Chicago O'Hare Int'l | ORD | terminal | ORD
major | Chicago O'Hare Int'l | ORD | terminal | ORD
and they're still in some projection using meters:
openmaptiles=# select ST_AsGeoJson(geometry) from ne_50m_airports where name like '%Dallas%' limit 1;
st_asgeojson
---------------------------------------------------------------------
{"type":"Point","coordinates":[-10802484.6943607,3882058.10739509]}
CAUTION: the download script uses asyncio.run, which is a Python3.7 addition
I think it has to be manually downloaded.
INOP: + download-osm planet -o data/nc.osm.pbf
1) get a bouning box of NC
http://bboxfinder.com/#34.633208,-82.265625,36.703660,-76.464844
2) crop the (gigantic) planet.osm.pbf to something smaller...
osmconvert:
-b=<x1>,<y1>,<x2>,<y2>
If you want to limit the geographical region, you can define a bounding box.
To do this, enter the __southwestern and the northeastern corners__ of that area.
which makes it:
osmconvert planet-latest.osm.pbf -b=-82.265625,34.633208,-76.464844,36.703660 -o=nc.osm.pbf
tools/imposm3/bin/imposm import -config /home/wendell/openmaptiles/wdb-map-gen/config/generated/imposm.json \
-overwritecache -read data/nc4.osm.pbf -deployproduction -write -diff
which says (I have no idea what diff mode is):
Importing in diff mode
+ tools/imposm3/bin/imposm import -config /home/wendell/openmaptiles/wdb-map-gen/config/generated/imposm.json -overwritecache -read data/nc4.osm.pbf -deployproduction -write -diff
[2021-10-03T19:37:34Z] 0:00:00 [info] removing existing cache /home/wendell/openmaptiles/wdb-map-gen/data/imposm3_cache
[2021-10-03T19:37:34Z] 0:00:00 [step] Starting: Imposm
[2021-10-03T19:37:34Z] 0:00:00 [step] Starting: Reading OSM data
[2021-10-03T19:37:34Z] 0:00:00 [info] reading data/nc4.osm.pbf with data till 2021-09-05 23:59:59 +0000 UTC
[2021-10-03T19:37:58Z] 0:00:24 [progress] 24s C: 0/s (38065496) N: 0/s (444825) W: 0/s (3673811) R: 0/s (11161)
[2021-10-03T19:37:58Z] 0:00:24 [step] Finished: Reading OSM data in 24.295867171s
[2021-10-03T19:37:58Z] 0:00:24 [step] Starting: Importing OSM data
[2021-10-03T19:37:58Z] 0:00:24 [step] Starting: Writing OSM data
[2021-10-03T19:38:58Z] 0:01:24 [progress] 1m0s C: 0/s ( 0.0%) N: 0/s ( 0.0%) W: 52100/s (72.1%) R: 190/s (100.0%)
[2021-10-03T19:39:16Z] 0:01:41 [progress] 1m17s C: 0/s ( 0.0%) N: 0/s (100.0%) W: 72300/s (100.0%) R: 190/s (100.0%)
[2021-10-03T19:39:16Z] 0:01:42 [step] Finished: Writing OSM data in 1m17.302576866s
[2021-10-03T19:39:16Z] 0:01:42 [step] Starting: Creating generalized tables
[2021-10-03T19:39:16Z] 0:01:42 [step] Starting: Generalizing osm_aerialway_linestring into osm_aerialway_linestring_gen_z12
[2021-10-03T19:39:16Z] 0:01:42 [step] Starting: Generalizing osm_aeroway_linestring into osm_aeroway_linestring_gen_z12
[2021-10-03T19:39:16Z] 0:01:42 [step] Starting: Generalizing osm_railway_linestring into osm_railway_linestring_gen_z12
[2021-10-03T19:39:16Z] 0:01:42 [step] Starting: Generalizing osm_border_disp_relation into osm_border_disp_linestring
[2021-10-03T19:39:16Z] 0:01:42 [step] Starting: Generalizing osm_park_polygon into osm_park_polygon_gen_z13
...
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry indices
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_border_disp_relation
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_peak_point
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_island_point
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating OSM id index on osm_railway_linestring_gen_z11
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating OSM id index on osm_shipway_linestring_gen_z12
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_building_polygon
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating OSM id index on osm_border_disp_linestring_gen_z12
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_landuse_polygon
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_country_point
[2021-10-03T19:39:21Z] 0:01:47 [step] Starting: Creating geometry index on osm_shipway_linestring
...
[2021-10-03T19:39:40Z] 0:02:06 [step] Starting: Rotating tables
[2021-10-03T19:39:40Z] 0:02:06 [info] Rotating osm_route_member from import -> public -> backup
[2021-10-03T19:39:40Z] 0:02:06 [info] Rotating osm_island_polygon from import -> public -> backup
[2021-10-03T19:39:40Z] 0:02:06 [info] Rotating osm_aerodrome_label_point from import -> public -> backup
[2021-10-03T19:39:40Z] 0:02:06 [info] Rotating osm_city_point from import -> public -> backup
...
[2021-10-03T19:39:40Z] 0:02:06 [step] Finished: Rotating tables in 173.10067ms
[2021-10-03T19:39:40Z] 0:02:06 [step] Finished: Imposm in 2m6.507305141s
+ set +x
====> : End importing Planet OpenStreetMap data: data/nc4.osm.pbf -> imposm3[./config/mapping.yaml] -> PostgreSQL
ok, not all that long (for N.C. alone), let's take a look:
openmaptiles=# \dt *railway*
List of relations
Schema | Name | Type | Owner
--------+--------------------------------+-------+-------
public | osm_railway_linestring | table | tiles
public | osm_railway_linestring_gen_z10 | table | tiles
public | osm_railway_linestring_gen_z11 | table | tiles
public | osm_railway_linestring_gen_z12 | table | tiles
public | osm_railway_linestring_gen_z8 | table | tiles
public | osm_railway_linestring_gen_z9 | table | tiles
(6 rows)
and:
openmaptiles=# \d osm_railway_linestring;
Table "public.osm_railway_linestring"
Column | Type | Collation | Nullable | Default
------------+---------------------------+-----------+----------+----
id | integer | | not null | nextval(...
osm_id | bigint | | not null |
railway | character varying | | |
ref | character varying | | |
network | character varying | | |
z_order | integer | | |
layer | integer | | |
level | integer | | |
indoor | boolean | | |
name | character varying | | |
name_en | character varying | | |
name_de | character varying | | |
tags | hstore | | |
short_name | character varying | | |
is_tunnel | boolean | | |
is_bridge | boolean | | |
is_ramp | boolean | | |
is_ford | boolean | | |
is_oneway | smallint | | |
is_area | boolean | | |
service | character varying | | |
usage | character varying | | |
geometry | geometry(LineString,3857) | | |
Indexes:
"osm_railway_linestring_pkey" PRIMARY KEY, btree (osm_id, id)
"osm_railway_linestring_geom" gist (geometry)
and:
openmaptiles=# select railway, network, name, name_en, short_name, service, usage from osm_railway_linestring where name != '' limit 55;
railway | name | service | usage
-----------+---------------------------------------------+---------+------------
rail | Pulaski District | | main
rail | Lancaster and Chester | | branch
rail | Edgemoor and Manetta Railway | spur |
rail | Lancaster and Chester Railroad | | branch
rail | Lancaster and Chester | | branch
rail | Wallace Siding | siding |
rail | Monroe Subdivision | | main
...
rail | High Point, Thomasville and Denton Railroad | |
rail | High Point, Thomasville and Denton Railroad | |
rail | Winston-Salem District | | main
rail | Winston-Salem District | | main
rail | Duke Branch | spur |
rail | Charlotte Subdivision | | main
rail | Aberdeen and Rockfish Railroad | | branch
...
rail | Winston-Salem Southbound Railway | | branch
rail | North Carolina and Virginia Railroad | | branch
rail | North Carolina and Virginia Railroad | | branch
rail | North Carolina and Virginia Railroad | | branch
but still, units are meters:
openmaptiles=# select ST_AsGeoJson(geometry) from osm_railway_linestring where name like '%Denton%';
{"type":"LineString","coordinates":[[-8913409.74216988,4284236.53777683],...
this seemed useful:
> vim ./osm_pkgs/openmaptiles-tools/build/scripts-3.6/import-wikidata
if __name__ == '__main__':
#wt: asyncio.run(main(docopt(__doc__, version=openmaptiles.__version__)))
loop = asyncio.get_event_loop()
loop.run_until_complete(main(docopt(__doc__, version=openmaptiles.__version__)))
(vmpython) wendell@opal4:~/openmaptiles/osm_pkgs/openmaptiles-tools
> python3 setup.py build
running build
running build_py
running build_scripts
(vmpython) wendell@opal4:~/openmaptiles/osm_pkgs/openmaptiles-tools
> python3 setup.py install
running install
after edits for async.run as above:
./load_wikidata.sh
which seemed to work:
Found 13 tables with the 'tags' hstore fields
* osm_aerodrome_label_point
* osm_city_point
* osm_continent_point
* osm_country_point
* osm_island_point
* osm_island_polygon
* osm_marine_point
* osm_peak_point
* osm_poi_point
* osm_poi_polygon
* osm_state_point
* osm_water_polygon
* osm_waterway_linestring
Cache file /home/wendell/openmaptiles/wdb-map-gen/data/cache/wikidata-cache.json does not exist, will create
Searching for Wikidata IDs in 13 tables...
Found 1,674 Wikidata IDs to load.
Dropping and Re-creating wd_names table...
NOTICE: table "wd_names" does not exist, skipping @ None
Query Wikidata Query Service for 1,674 IDs...
Inserted 1,669 Wikidata IDs from WDQS into wd_names table
Resolving 5 possible Wikidata redirects...
5 out of 5 Wikidata ID redirects were found
Querying Wikidata for 5 redirect IDs...
Inserted 5 Wikidata IDs from WDQS into wd_names table
Saving 1,674 items to cache /home/wendell/openmaptiles/wdb-map-gen/data/cache/wikidata-cache.json
needed to edit for "-p $POSTGRES_PASS" in a couple of places
vim load_sql.sh
openmaptiles=# create database tiles; CREATE DATABASE openmaptiles=# \q (vmpython) wendell@opal4:~/openmaptiles
psql --port 55432 tilespsql (12.8) Type "help" for help.
tiles=# \d Did not find any relations. tiles=# create extension postgis; CREATE EXTENSION tiles=# create extension hstore; CREATE EXTENSION tiles=# grant all privileges on database tiles to tiles ; GRANT
ZZ then:
./load_sql.sh
Single-THREADED
-- Importing: /home/wendell/openmaptiles/wdb-map-gen/config/sql/CleanNumeric.sql --
CREATE FUNCTION
-- Importing: /home/wendell/openmaptiles/wdb-map-gen/config/sql/Z.sql --
CREATE FUNCTION
-- Importing: /home/wendell/openmaptiles/wdb-map-gen/config/sql/zzz_language.sql --
CREATE FUNCTION
CREATE FUNCTION
<lots of stuff>
then: CREATE FUNCTION psql:/home/wendell/openmaptiles/wdb-map-gen/data/sql/tileset.sql:7788: NOTICE: Finished layer aerodrome_label DO psql: error: FATAL: database "tiles" does not exist
SO, a database 'tiles' was created, BUT IT WAS NOT POPULATED!!!
and load_sql seemed to have completed
attempts to find out data format:
> ogrinfo -al dallas.osm.pbf
INFO: Open of `dallas.osm.pbf'
using driver `OSM' successful.
Layer name: points
Geometry: Point
Feature Count: -1
Extent: (-98.300000, 32.000000) - (-95.520000, 33.850000)
Layer SRS WKT:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
osm_id: String (0.0)
name: String (0.0)
barrier: String (0.0)
highway: String (0.0)
ref: String (0.0)
address: String (0.0)
is_in: String (0.0)
place: String (0.0)
man_made: String (0.0)
other_tags: String (0.0)
OGRFeature(points):26451915
osm_id (String) = 26451915
highway (String) = motorway_junction
other_tags (String) = "noref"=>"yes"
POINT (-96.9963659 32.9878311)
and
> ogrinfo dallas.osm.pbf lines | less
INFO: Open of `dallas.osm.pbf'
using driver `OSM' successful.
Layer name: lines
Geometry: Line String
Feature Count: -1
Extent: (-98.300000, 32.000000) - (-95.520000, 33.850000)
Layer SRS WKT:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
osm_id: String (0.0)
name: String (0.0)
highway: String (0.0)
waterway: String (0.0)
aerialway: String (0.0)
barrier: String (0.0)
man_made: String (0.0)
z_order: Integer (0.0)
other_tags: String (0.0)
OGRFeature(lines):4342988
osm_id (String) = 4342988
name (String) = Sam Rayburn Tollway
highway (String) = motorway
z_order (Integer) = 9
other_tags (String) = "lanes"=>"2","maxspeed"=>"70 mph","old_ref"=>"Toll 121",...
LINESTRING (-96.9974646 32.9882702,-97.0004718 32.9894883,...
OGRFeature(lines):4343083
osm_id (String) = 4343083
name (String) = Sam Rayburn Tollway
highway (String) = motorway
z_order (Integer) = 9
other_tags (String) = "lanes"=>"3","maxspeed"=>"70 mph","old_ref"=>"Toll 121",...
LINESTRING (-97.0060288 32.9905785,-97.0056765 32.990593,-97.0052949 32.9905893,...