Python download files from web page

Find centralized, trusted content and collaborate around the technologies you use most. Connect and share knowledge within a single location that is structured and easy to search. I would use urljoin to join the url and you can use just the xpath to get the hrefs, you don't need to call find:. If you want to make asynchronous you could utilise the grequests lib:. This may be a better question for Code Review. In short, your code is fine. If anything, you might want to use more lines.

Here's my attempt at cleaning it up some I'm not aware of any module that will combine some of these steps. Your code is relatively readable and I don't see any inefficiencies. In summary, I think the biggest mistake is thinking that using less lines would improve your code at least in this case. Stack Overflow for Teams — Collaborate and share knowledge with a private group.

Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Asked 5 years, 8 months ago. Active 5 years, 8 months ago. Viewed times. I am running below code to download all files in a webpage: import os import urllib from lxml import html def main : os.

Add code to use the url to extract the hostname, don't hardcode it. If your sole goal is to have less lines, there isn't much else beside maybe using some more modern libraries to aid the work like python-requests and BeautifulSoup 4 or even some full toolset like Scrapy.

Add a comment. Active Oldest Votes. A fixed chunk will be loaded each time while r. All the archives of this lecture are available here. So, we first scrape the webpage to extract all video links and then download the videos one by one. It would have been tiring to download each video manually. In this example, we first crawl the webpage to extract all the links and then download videos.

This is a browser-independent method and much faster! One can simply scrape a web page to get all the file URLs on a webpage and hence, download all files in a single command- Implementing Web Scraping in Python with BeautifulSoup This blog is contributed by Nikhil Kumar. If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute. See your article appearing on the GeeksforGeeks main page and help other Geeks. Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

Skip to content. Change Language. Related Articles. Table of Contents. Save Article. Improve Article. Like Article. Saving received content as a png file in. URL of the archive web-page which provides link to. It would have been tiring to. In this example, we first crawl the webpage to extract.