Bulk Lidar Download on Digital Coast


Sometimes you just need all the lidar data you can lay your hands on. While our Data Access Viewer lets you manipulate and customize the data before you get it, there are limits to how much it will process at a time. Thus we set up simple access to the files without processing or size limits. There are a few ways to get to the bulk download. If you’re familiar with our site, skip down to the section on uGet for the new information.

Data Access Viewer (DAV)

The DAV system lets you do a search on a map to find the data sets, which can be very handy. The general idea is to use the the pencil tool in the upper left to draw a box around your area of interest and add it to the cart for custom processing. If what you want is too big (the add to cart doesn’t light up) and you don’t want to break it into pieces, there is a BULK DOWNLOAD link in the search results that will take you to the page for that data set. By the way, if you aren’t looking to get tons of data but just need a particular area, the DAV system is the way to go. Check this earlier post on the DAV features and use.

Metadata

You may have found information about the data set by first finding the metadata. All of our metadata is on the InPort system and you can search there. Our lidar metadata will generally have two links for data distribution. One is to the DAV system. The other is to the bulk download. It should have a description mentioning bulk download and the URL should include a geoid name (e.g. geoid12b).

The Secret Page

There is a web page with a big table of data sets. I often call this the secret page. It’s not really secret, but it is a page that isn’t really linked to from our web site. It was originally put in place so you would get something if you navigated up from one of the data sets, but has turned out to be quite useful. In addition to the title, the table contains the links to the metadata, ftp access, https access, the DAV system, a KMZ, and the tile index shapefile.

Downloading with Wget

Most of the above links lead you to the https access and the page for the dataset provides links to all the files and instructions to download the whole data set using the wget command (also compiled for Windows wget). This works well for a lot of people, however, it can be a stumbling block for Windows users that aren’t used to the command line. It also has the unfortunate problem that Windows Powershell has an alias for wget that isn’t really wget, adding to the confusion.

Downloading with uGet

Since people had trouble with wget, we’ve now added some support for uGet. You still have to download and install it, but it does have a GUI interface. It’s also the program the National Map references for their downloads, so you may already be familiar with it. What we added was a text file that lists all the URLs for the data set. It shows on the page as the URL list. Here are the steps you’ll take with uget to download using that file.

  • Download the URL list file. You’ll probably need to right click on the link and save as. The name of the file will include a number that identifies the data set. That’s the same number you’ll find in the ID column of the secret page.
Example bulk download web page showing the URL list link.
Example bulk download web page showing the URL list link.
  • In uGet, open the file tab. One of your choices will be Batch Downloads, which opens another menu when you hover over it. Pick the Text file import (*.txt) selection. In the dialog box that opens, select the URL list file you just saved.
Text file import menu in uGet.
  • uGet will read the file and provide a list of all the URLs to download. You can uncheck things you don’t want. There is a filter button if you want to filter by file extension.
  • Before you hit the OK button and start downloading, make sure you hit the Forward button. This will let you pick the folder you want things downloaded to. I missed that the first time through and downloaded a data set into my Windows home folder. Lesson learned.
The forward button. Don’t miss this.
  • Now you can hit the OK button and start the download.

All done

That’s it, you’re all done. Whether you used uGet with the new URL list or good old wget or went the custom route, I hope you’ve gotten your lidar data. If you’ve got other suggestions that would make it easier to download the data, please let us know. If you only want the data within a polygon, you might want to check “Getting lidar data with a polygon“.

2 comments

  1. This article and the Getting lidar data with a polygon were both extremely helpful. My recommendation would be to add an ftp url list in the ftp index page. The one that appears for the 2015-2017 USGS Lidar DEM: Puerto Rico is the http url list. I used wget, and it did not connect, but when I replaced the http part of the url with the corresponding ftp path in the url, it worked and started to download.

    Like

Leave a Reply. Comments are moderated.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.