Как утащить что угодно с любого сайта

Сего­дняш­ний про­ект послу­жит осно­вой мно­гих наших даль­ней­ших про­грамм. Мы научим­ся соби­рать с сай­тов любые дан­ные, кото­рые нам нужны. 

У нас есть рабо­чий про­ект на цепях Мар­ко­ва. Цепи Мар­ко­ва — это неслож­ный алго­ритм, кото­рый ана­ли­зи­ру­ет соче­та­е­мость слов в задан­ном тек­сте и выда­ёт новый текст на осно­ве ста­ро­го. Похо­же на рабо­ту ней­ро­нок, но на самом деле это про­сто пере­бор слов и бес­смыс­лен­ное их сочетание. 

Для рабо­ты наших пер­вых про­ек­тов на цепях Мар­ко­ва мы ска­ча­ли кни­гу с рас­ска­за­ми Чехо­ва. Про­грам­ма ана­ли­зи­ру­ет соче­та­е­мость чехов­ских слов и выда­ёт текст в чехов­ском духе (хотя и бессмысленный).

Но что, если мы хотим сде­лать текст не в духе Чехо­ва, а в духе жур­на­ла «Код»? Или в духе какого-нибудь издания-иноагента? Или сде­лать гене­ра­тор ста­тей в духе какого-нибудь блогера? 

Реше­ние — напи­сать про­грам­му, кото­рая посмот­рит на сай­те наши ста­тьи и выта­щит отту­да весь зна­чи­мый текст. Един­ствен­ное, что для это­го пона­до­бит­ся, — спи­сок ссы­лок на ста­тьи, но мы их уже собра­ли, когда дела­ли про­ект с гада­ни­ем на ста­тьях Кода.

Логика работы

Про­грам­ма будет рабо­тать на Python на локаль­ной машине. Алгоритм:

  1. Под­клю­ча­ем нуж­ные библиотеки.
  2. Гото­вим пере­мен­ную со спис­ком сай­тов для обхода.
  3. Захо­дим на пер­вый сайт и полу­ча­ем отту­да исход­ный код.
  4. Ука­зы­ва­ем, отку­да брать текст.
  5. Сохра­ня­ем най­ден­ный текст в отдель­ном файле.
  6. Повто­ря­ем цикл до тех пор, пока не прой­дём все стра­ни­цы из списка.

👉 Глав­ное в таких про­ек­тах — знать струк­ту­ру содер­жи­мо­го стра­ни­цы и пони­мать, где имен­но и в каких тегах нахо­дят­ся нуж­ные для вас данные. 

Что­бы было про­ще, на стар­те сде­ла­ем про­грам­му, кото­рая соби­ра­ет назва­ния стра­ниц. Как осво­им­ся — сде­ла­ем что посложнее.

Изучаем исходный код страницы

Преж­де чем зани­мать­ся пар­син­гом (сбо­ром) со стра­ни­цы чего угод­но, нуж­но выяс­нить, где это лежит и в какой коди­ров­ке. Мы зна­ем, что все ста­тьи Кода созда­ны по одно­му и тому же шаб­ло­ну, поэто­му нам доста­точ­но посмот­реть, как устро­е­на одна, что­бы понять их все. 

Смот­рим исход­ный код любой нашей ста­тьи. Нас инте­ре­су­ют два момен­та — коди­ров­ка стра­ни­цы и тег <title>. Нам нуж­но убе­дить­ся, что в этом теге про­пи­са­но название. 

Сна­ча­ла кодировка:

Как утащить что угодно с любого сайта

Эта строч­ка озна­ча­ет, что стра­ни­ца рабо­та­ет с коди­ров­кой UTF-8. Запом­ним это.

Теперь про­ли­сты­ва­ем исход­ный код ниже и нахо­дим тег <title> — имен­но он отве­ча­ет за заго­ло­вок стра­ни­цы. Убеж­да­ем­ся, что он есть и с ним всё в порядке:

Как утащить что угодно с любого сайта

Библиотеки для работы

В про­ек­те нам пона­до­бят­ся две биб­лио­те­ки: urllib и BeautifulSoup.

Пер­вая отве­ча­ет за доступ к стра­ни­цам по их адре­су, при­чём отту­да нам будет нуж­на толь­ко одна коман­да urlopen().read — она отправ­ля­ет­ся по ука­зан­но­му адре­су и полу­ча­ет весь исход­ный код страницы.

Вто­рая биб­лио­те­ка вхо­дит в состав боль­шой биб­лио­те­ки bs4 — в ней уже собра­ны все коман­ды для пар­син­га исход­но­го HTML-кода и раз­бо­ра тегов. Что­бы уста­но­вить bs4, запус­ка­ем тер­ми­нал и пишем:

pip3 install bs4

Как утащить что угодно с любого сайта

Пишем код

Сна­ча­ла под­клю­чим все нуж­ные библиотеки:

# подключаем urlopen из модуля urllib

from urllib.request import urlopen

# подключаем библиотеку BeautifulSoup

from bs4 import BeautifulSoup

Теперь объ­явим спи­сок стра­ниц, кото­рые нуж­но посе­тить и забрать отту­да заго­лов­ки. Мы уже соста­ви­ли такой спи­сок для про­ек­та с гада­ни­ем на ста­тьях Кода, поэто­му про­сто возь­мём его отту­да и адап­ти­ру­ем под Python:

Объявляем массив со статьями

url = [

"https://thecode.media/is-not-defined-jquery/",

"https://thecode.media/arduino-projects-2/",

"https://thecode.media/10-raspberry/",

"https://thecode.media/easy-css/",

"https://thecode.media/to-be-front/",

"https://thecode.media/cryptex/",

"https://thecode.media/ali-coders/",

"https://thecode.media/po-glandy/",

"https://thecode.media/megaexcel/",

"https://thecode.media/chat-bot-generators/",

"https://thecode.media/wifi/",

"https://thecode.media/andri-oxa/",

"https://thecode.media/free-hosting/",

"https://thecode.media/hotwheels/",

"https://thecode.media/do-not-disturb/",

"https://thecode.media/dyno-ai/",

"https://thecode.media/snake-ai/",

"https://thecode.media/leet/",

"https://thecode.media/ninja/",

"https://thecode.media/supergirl/",

"https://thecode.media/vpn/ ",

"https://thecode.media/what-is-wordpress/",

"https://thecode.media/hardware/",

"https://thecode.media/division/",

"https://thecode.media/nuggets/",

"https://thecode.media/binary-notation/",

"https://thecode.media/bootstrap/",

"https://thecode.media/chat-bot/",

"https://thecode.media/myadblock3000/",

"https://thecode.media/trello/",

"https://thecode.media/python-time/",

"https://thecode.media/editor/",

"https://thecode.media/timer/",

"https://thecode.media/intro-bootstrap/",

"https://thecode.media/php-form/",

"https://thecode.media/hr-quiz/",

"https://thecode.media/c-sharp/",

"https://thecode.media/showtime/",

"https://thecode.media/uchtel-rasskazhi/",

"https://thecode.media/sshhhh/",

"https://thecode.media/marry-me-python/",

"https://thecode.media/haters-gonna-code/",

"https://thecode.media/speed-css/",

"https://thecode.media/fired/",

"https://thecode.media/zabuhal/",

"https://thecode.media/est-tri-shkatulki/",

"https://thecode.media/milk-that/",

"https://thecode.media/binary-mouse/",

"https://thecode.media/bowling/",

"https://thecode.media/dealership/",

"https://thecode.media/best-seller/",

"https://thecode.media/hr/",

"https://thecode.media/no-comments/",

"https://thecode.media/drakoni-yajca/",

"https://thecode.media/who-is-who/",

"https://thecode.media/get-a-room/",

"https://thecode.media/alps/",

"https://thecode.media/handshake/",

"https://thecode.media/choose-life/",

"https://thecode.media/high-voltage/",

"https://thecode.media/spy/",

"https://thecode.media/squirrelrrel/",

"https://thecode.media/so-agile/",

"https://thecode.media/wedding/",

"https://thecode.media/supper/",

"https://thecode.media/le-tarakan/",

"https://thecode.media/batareyki-besyat/",

"https://thecode.media/dr_jekyll/",

"https://thecode.media/everybody_lies/",

"https://thecode.media/electrician/",

"https://thecode.media/einstein/",

"https://thecode.media/bugz/",

"https://thecode.media/needforspeed/",

"https://thecode.media/be-smart/",

"https://thecode.media/bot-online/",

"https://thecode.media/microb/",

"https://thecode.media/jquery/",

"https://thecode.media/split-screen/",

"https://thecode.media/calculus/",

"https://thecode.media/big-data-sales/",

"https://thecode.media/ambient/",

"https://thecode.media/fatality/",

"https://thecode.media/biggest-loser/",

"https://thecode.media/wifi/",

"https://thecode.media/nosock/",

"https://thecode.media/variables/",

"https://thecode.media/start_python/",

"https://thecode.media/i-gonna-code/",

"https://thecode.media/sigi-est/",

"https://thecode.media/nes-game/",

"https://thecode.media/live-view/",

"https://thecode.media/remote/",

"https://thecode.media/arduino-code/",

"https://thecode.media/horses/",

"https://thecode.media/runinstein/",

"https://thecode.media/wp-template/",

"https://thecode.media/tilda/",

"https://thecode.media/todo/",

"https://thecode.media/telebot/",

"https://thecode.media/summator-2/",

"https://thecode.media/get-rich-coding/",

"https://thecode.media/content-manager/",

"https://thecode.media/vzrosly-stal/",

"https://thecode.media/py-install/",

"https://thecode.media/quantum/",

"https://thecode.media/dns/",

"https://thecode.media/practicum/",

"https://thecode.media/react/",

"https://thecode.media/1september/",

"https://thecode.media/summator/",

"https://thecode.media/vds/",

"https://thecode.media/made-in-china/",

"https://thecode.media/bar/",

"https://thecode.media/zodiac/",

"https://thecode.media/crc32/",

"https://thecode.media/css-links/",

"https://thecode.media/oop_battle/",

"https://thecode.media/be-combo/",

"https://thecode.media/unity/",

"https://thecode.media/data-science/",

"https://thecode.media/junior/",

"https://thecode.media/qc/",

"https://thecode.media/be-middle/",

"https://thecode.media/senior/",

"https://thecode.media/teamlead/",

"https://thecode.media/frontend/",

"https://thecode.media/lift/",

"https://thecode.media/be-fuzzy/",

"https://thecode.media/best-2020/",

"https://thecode.media/git/",

"https://thecode.media/stt-cloud/",

"https://thecode.media/matrix-pills/",

"https://thecode.media/na-stile/",

"https://thecode.media/no-coffee/",

"https://thecode.media/framelibs/",

"https://thecode.media/children/",

"https://thecode.media/balls-possibly/",

"https://thecode.media/le-meduza/",

"https://thecode.media/electricity/",

"https://thecode.media/tailored-swift/",

"https://thecode.media/objective/",

"https://thecode.media/host/",

"https://thecode.media/go-public/",

"https://thecode.media/how-internet-works-1/",

"https://thecode.media/domain/",

"https://thecode.media/this-is-object/",

"https://thecode.media/ole-ole-ole/",

"https://thecode.media/thousand/",

"https://thecode.media/average/",

"https://thecode.media/stt-python/",

"https://thecode.media/ping-pong/",

"https://thecode.media/pygames/",

"https://thecode.media/odobreno/",

"https://thecode.media/qwerty123/",

"https://thecode.media/neurocorrector/",

"https://thecode.media/neuro-cam/",

"https://thecode.media/10-jquery/",

"https://thecode.media/repeat/",

"https://thecode.media/assembler/",

"https://thecode.media/sublime-one-love/",

"https://thecode.media/zloy/",

"https://thecode.media/mariya-ivanovna/",

"https://thecode.media/ruby/",

"https://thecode.media/electron-password/",

"https://thecode.media/plane/",

"https://thecode.media/glitch/",

"https://thecode.media/security/",

"https://thecode.media/stupid-2019/",

"https://thecode.media/jquery-search/",

"https://thecode.media/pimp-my-pass/",

"https://thecode.media/text-ultimate/",

"https://thecode.media/hurry/",

"https://thecode.media/siri/",

"https://thecode.media/zero-cool/",

"https://thecode.media/small-talk/",

"https://thecode.media/die-hard/",

"https://thecode.media/le-piton/",

"https://thecode.media/hr-code/",

"https://thecode.media/nano-code/",

"https://thecode.media/the_question/",

"https://thecode.media/godlike/",

"https://thecode.media/be-logic/",

"https://thecode.media/snake-js/",

"https://thecode.media/be-mobile/",

"https://thecode.media/baboolya/",

"https://thecode.media/timelag/",

"https://thecode.media/doors/",

"https://thecode.media/phone-code/",

"https://thecode.media/snake-arduino/",

"https://thecode.media/css-intro/",

"https://thecode.media/le-timer/",

"https://thecode.media/oop_battle/",

"https://thecode.media/good-morning/",

"https://thecode.media/study-bot/",

"https://thecode.media/python-bot/",

"https://thecode.media/robot-quiz/",

"https://thecode.media/hacking-quiz/",

"https://thecode.media/lulz-quiz/",

"https://thecode.media/hard-quiz/",

"https://thecode.media/torrent/",

"https://thecode.media/travel/",

"https://thecode.media/le-snob/",

"https://thecode.media/no-spagetti/",

"https://thecode.media/house/",

"https://thecode.media/cryptorush/",

"https://thecode.media/coronarelax/",

"https://thecode.media/pure/",

"https://thecode.media/c-cpp/",

"https://thecode.media/machine-loving/",

"https://thecode.media/orwell/",

"https://thecode.media/darknet/",

"https://thecode.media/ai/",

"https://thecode.media/oop-class/",

"https://thecode.media/cookie/",

"https://thecode.media/malware/",

"https://thecode.media/ftp/",

"https://thecode.media/html/",

"https://thecode.media/java/",

"https://thecode.media/php-haters/",

"https://thecode.media/tor/",

"https://thecode.media/crack-safe/",

"https://thecode.media/epidemic/",

"https://thecode.media/hash-brown/",

"https://thecode.media/java-js/",

"https://thecode.media/js-types/",

"https://thecode.media/losers/",

"https://thecode.media/ssl/",

"https://thecode.media/uncaughtsyntaxerror-unexpected-identifier/",

"https://thecode.media/uncaughtsyntaxerror-unexpected-token/",

"https://thecode.media/uncaughttyperrror-cannot-read-property/",

"https://thecode.media/mobile-dev/",

"https://thecode.media/verevka/",

"https://thecode.media/speed/",

"https://thecode.media/buckwheat/",

"https://thecode.media/distance/",

"https://thecode.media/node-js/",

"https://thecode.media/pascal/",

"https://thecode.media/ill-be-clean/",

"https://thecode.media/to-be-back/",

"https://thecode.media/replaceable/",

"https://thecode.media/code-review/",

"https://thecode.media/gasoline/",

"https://thecode.media/to-be-test/",

"https://thecode.media/scala/",

"https://thecode.media/row-power/",

"https://thecode.media/percent/",

"https://thecode.media/с/",

"https://thecode.media/things/",

"https://thecode.media/prof-newsletter/",

"https://thecode.media/backend/",

"https://thecode.media/immortal-pong/",

"https://thecode.media/blind/",

"https://thecode.media/go-faster/",

"https://thecode.media/cpp/",

"https://thecode.media/uncaught-syntaxerror-unexpected-end-of-input/",

"https://thecode.media/stress-quiz/",

"https://thecode.media/secret-pong/",

"https://thecode.media/override/",

"https://thecode.media/whg/",

"https://thecode.media/profit/",

"https://thecode.media/memas/",

"https://thecode.media/digital-sound/",

"https://thecode.media/api/",

"https://thecode.media/be-math-2/",

"https://thecode.media/backup/",

"https://thecode.media/backup-master/",

"https://thecode.media/glvrd/",

"https://thecode.media/id/",

"https://thecode.media/uncaught-syntaxerror-missing-after-argument-list/",

"https://thecode.media/ex-startup/",

"https://thecode.media/doom-everywhere/",

"https://thecode.media/template-one/",

"https://thecode.media/david-roganov/",

"https://thecode.media/spacex/",

"https://thecode.media/webstorm/",

"https://thecode.media/json/",

"https://thecode.media/treger/",

"https://thecode.media/ya-blitz/",

"https://thecode.media/radius/",

"https://thecode.media/xhr/",

"https://thecode.media/treger2/",

"https://thecode.media/raidemption/",

"https://thecode.media/chief-technical-officer/",

"https://thecode.media/summary/",

"https://thecode.media/ex-wallpaper/",

"https://thecode.media/soap/",

"https://thecode.media/decompose/",

"https://thecode.media/desc/",

"https://thecode.media/sprint/",

"https://thecode.media/bye-or-die/",

"https://thecode.media/who-win/",

"https://thecode.media/vladimir-olokhtonov/",

"https://thecode.media/lossless/",

"https://thecode.media/parse/",

"https://thecode.media/typeerror-is-not-an-abject/",

"https://thecode.media/backup-me/",

"https://thecode.media/stress-test/",

"https://thecode.media/syntaxerror-missing-formal-parameter/",

"https://thecode.media/start-fast/",

"https://thecode.media/halkechev/",

"https://thecode.media/halkechev2/",

"https://thecode.media/le-design/",

"https://thecode.media/syntaxerror-missing-after-property-id/",

"https://thecode.media/attrb-mthd/",

"https://thecode.media/headphones/",

"https://thecode.media/active-noise-cancelling/",

"https://thecode.media/remote-work-quiz/",

"https://thecode.media/garbage/",

"https://thecode.media/ubuntu-linux/",

"https://thecode.media/trie/",

"https://thecode.media/func/",

"https://thecode.media/laravel/",

"https://thecode.media/save-json/",

"https://thecode.media/syntaxerror-missing-after-formal-parameters/",

"https://thecode.media/recursion/",

"https://thecode.media/haskell/",

"https://thecode.media/gen/",

"https://thecode.media/db/",

"https://thecode.media/boosting/",

"https://thecode.media/pavel-sviridov/",

"https://thecode.media/mnogo/",

"https://thecode.media/sokr/",

"https://thecode.media/dbsm/",

"https://thecode.media/pik-balmera/",

"https://thecode.media/kanban/",

"https://thecode.media/check-list/",

"https://thecode.media/text-quiz/",

"https://thecode.media/mysql/",

"https://thecode.media/mysql/",

"https://thecode.media/rust/",

"https://thecode.media/manage-this/",

"https://thecode.media/altshuller/",

"https://thecode.media/interview/",

"https://thecode.media/fotorama/",

"https://thecode.media/tetris/",

"https://thecode.media/ai-tetris/",

"https://thecode.media/scrum/",

"https://thecode.media/speed-two/",

"https://thecode.media/quick-share/",

"https://thecode.media/stack/",

"https://thecode.media/mobile-first/",

"https://thecode.media/nosql/",

"https://thecode.media/narazves/",

"https://thecode.media/oop-class-2/",

"https://thecode.media/design-first/",

"https://thecode.media/arcanoid/",

"https://thecode.media/donut/",

"https://thecode.media/casino/",

"https://thecode.media/heap/",

"https://thecode.media/rust/",

"https://thecode.media/float/",

"https://thecode.media/markdown/",

"https://thecode.media/books/",

"https://thecode.media/daniil-popov/",

"https://thecode.media/android-developer/",

"https://thecode.media/symbols/",

"https://thecode.media/oauth/",

"https://thecode.media/kotlin/",

"https://thecode.media/todo/",

"https://thecode.media/plotly/",

"https://thecode.media/no-digit-code/",

"https://thecode.media/asymmetric/",

"https://thecode.media/qi/",

"https://thecode.media/vernam/",

"https://thecode.media/vernam-js/",

"https://thecode.media/shtykov/",

"https://thecode.media/memory/",

"https://thecode.media/ark/",

"https://thecode.media/7-oshibok-na-sobesedovanii/",

"https://thecode.media/dh/",

"https://thecode.media/typescript/",

"https://thecode.media/stark/",

"https://thecode.media/crypto/",

"https://thecode.media/zapusk-2/",

"https://thecode.media/fingerprint/",

"https://thecode.media/puzzle/",

"https://thecode.media/python-time-2/",

"https://thecode.media/no-chance/",

"https://thecode.media/lossy/",

"https://thecode.media/1wire/",

"https://thecode.media/pasha-flipper/",

"https://thecode.media/perl/",

"https://thecode.media/alexey-vasilev/",

"https://thecode.media/viasat/",

"https://thecode.media/podcast/",

"https://thecode.media/copy-ya-ru/",

"https://thecode.media/mircrosd/",

"https://thecode.media/bash/",

"https://thecode.media/rotation/",

"https://thecode.media/css-grid/",

"https://thecode.media/train/",

"https://thecode.media/grid-2/",

"https://thecode.media/za-proezd/",

"https://thecode.media/grid-3/",

"https://thecode.media/david/",

"https://thecode.media/alien-vs-predator/",

"https://thecode.media/grid-portfolio/",

"https://thecode.media/podcast-lavka/",

"https://thecode.media/it-start-2/",

"https://thecode.media/anastasiya-nikulina/",

"https://thecode.media/linter/",

"https://thecode.media/bomberman/",

"https://thecode.media/5-linters/",

"https://thecode.media/lineynaya-algebra-vektory/",

"https://thecode.media/no-nda/",

"https://thecode.media/code-swap/",

"https://thecode.media/how-to-start/",

"https://thecode.media/referenceerror-invalid-left-hand-side-in-assignment/",

"https://thecode.media/vectors-operations/",

"https://thecode.media/oop-abstract/",

"https://thecode.media/leonov/",

"https://thecode.media/lucky-strike/",

"https://thecode.media/browser/",

"https://thecode.media/2020/",

"https://thecode.media/3d-stars/",

"https://thecode.media/anna-leonova/",

"https://thecode.media/cold-fusion/",

"https://thecode.media/normalize/",

"https://thecode.media/hotkey/",

"https://thecode.media/oven/",

"https://thecode.media/vim/",

"https://thecode.media/draw/",

"https://thecode.media/visual-studio-code/",

"https://thecode.media/tetris-2/",

"https://thecode.media/start-now/",

"https://thecode.media/lapsha-1/",

"https://thecode.media/cubism/",

"https://thecode.media/lapsha-2/",

"https://thecode.media/zerocode/",

"https://thecode.media/cat/",

"https://thecode.media/static/",

"https://thecode.media/komm/",

"https://thecode.media/path-js/",

"https://thecode.media/cloudly/",

"https://thecode.media/haters-gonna-code-2/",

"https://thecode.media/csp/",

"https://thecode.media/mitin-says-no/",

"https://thecode.media/hire-js/",

"https://thecode.media/tableau/",

"https://thecode.media/impossible/",

"https://thecode.media/csp-on/",

"https://thecode.media/maze/",

"https://thecode.media/mix/",

"https://thecode.media/le-beton/",

"https://thecode.media/3d-print/",

"https://thecode.media/5-and-a-half/",

"https://thecode.media/lineynaya-zavisimost-vektorov/",

"https://thecode.media/fast-m1/",

"https://thecode.media/ninja-run/",

"https://thecode.media/matrix-101/",

"https://thecode.media/arm-x86/",

"https://thecode.media/piano-js/",

"https://thecode.media/10-swift/",

"https://thecode.media/travel-plane/",

"https://thecode.media/extention/",

"https://thecode.media/obratnaya-matritsa/",

"https://thecode.media/svg/",

"https://thecode.media/freelance/",

"https://thecode.media/brat-2/",

"https://thecode.media/angular/",

"https://thecode.media/rgb/",

"https://thecode.media/10-go/",

"https://thecode.media/coffee/",

]

Теперь пере­би­ра­ем все эле­мен­ты это­го мас­си­ва в цик­ле, исполь­зуя всю мощь биб­лио­тек. Обра­ти­те вни­ма­ние на строч­ку, где мы полу­ча­ем исход­ный код стра­ни­цы — мы сра­зу кон­вер­ти­ру­ем его в нуж­ную коди­ров­ку, кото­рую выяс­ни­ли на преды­ду­щем этапе:

# открываем текстовый файл, куда будем добавлять заголовки
file = open("zag.txt", "a")

# перебираем все адреса из списка
for x in url:
    # получаем исходный код очередной страницы из списка
    html_code = str(urlopen(x).read(),'utf-8')
    # отправляем исходный код страницы на обработку в библиотеку
    soup = BeautifulSoup(html_code, "html.parser")

    # находим название страницы с помощью метода find()
    s = soup.find('title').text

    # выводим его на экран
    print(s)

    # сохраняем заголовок в файле и переносим курсор на новую строку
    file.write(s + '\n')

# закрываем файл
file.close()

Запус­ка­ем и смот­рим на результат:

Как утащить что угодно с любого сайта
Итоговый код

# подключаем urlopen из модуля urllib
from urllib.request import urlopen

# подключаем библиотеку BeautifulSout
from bs4 import BeautifulSoup

url = [
"https://thecode.media/is-not-defined-jquery/",
"https://thecode.media/arduino-projects-2/",
"https://thecode.media/10-raspberry/",
"https://thecode.media/easy-css/",
"https://thecode.media/to-be-front/",
"https://thecode.media/cryptex/",
"https://thecode.media/ali-coders/",
"https://thecode.media/po-glandy/",
"https://thecode.media/megaexcel/",
"https://thecode.media/chat-bot-generators/",
"https://thecode.media/wifi/",
"https://thecode.media/andri-oxa/",
"https://thecode.media/free-hosting/",
"https://thecode.media/hotwheels/",
"https://thecode.media/do-not-disturb/",
"https://thecode.media/dyno-ai/",
"https://thecode.media/snake-ai/",
"https://thecode.media/leet/",
"https://thecode.media/ninja/",
"https://thecode.media/supergirl/",
"https://thecode.media/vpn/ ",
"https://thecode.media/what-is-wordpress/",
"https://thecode.media/hardware/",
"https://thecode.media/division/",
"https://thecode.media/nuggets/",
"https://thecode.media/binary-notation/",
"https://thecode.media/bootstrap/",
"https://thecode.media/chat-bot/",
"https://thecode.media/myadblock3000/",
"https://thecode.media/trello/",
"https://thecode.media/python-time/",
"https://thecode.media/editor/",
"https://thecode.media/timer/",
"https://thecode.media/intro-bootstrap/",
"https://thecode.media/php-form/",
"https://thecode.media/hr-quiz/",
"https://thecode.media/c-sharp/",
"https://thecode.media/showtime/",
"https://thecode.media/uchtel-rasskazhi/",
"https://thecode.media/sshhhh/",
"https://thecode.media/marry-me-python/",
"https://thecode.media/haters-gonna-code/",
"https://thecode.media/speed-css/",
"https://thecode.media/fired/",
"https://thecode.media/zabuhal/",
"https://thecode.media/est-tri-shkatulki/",
"https://thecode.media/milk-that/",
"https://thecode.media/binary-mouse/",
"https://thecode.media/bowling/",
"https://thecode.media/dealership/",
"https://thecode.media/best-seller/",
"https://thecode.media/hr/",
"https://thecode.media/no-comments/",
"https://thecode.media/drakoni-yajca/",
"https://thecode.media/who-is-who/",
"https://thecode.media/get-a-room/",
"https://thecode.media/alps/",
"https://thecode.media/handshake/",
"https://thecode.media/choose-life/",
"https://thecode.media/high-voltage/",
"https://thecode.media/spy/",
"https://thecode.media/squirrelrrel/",
"https://thecode.media/so-agile/",
"https://thecode.media/wedding/",
"https://thecode.media/supper/",
"https://thecode.media/le-tarakan/",
"https://thecode.media/batareyki-besyat/",
"https://thecode.media/dr_jekyll/",
"https://thecode.media/everybody_lies/",
"https://thecode.media/electrician/",
"https://thecode.media/einstein/",
"https://thecode.media/bugz/",
"https://thecode.media/needforspeed/",
"https://thecode.media/be-smart/",
"https://thecode.media/bot-online/",
"https://thecode.media/microb/",
"https://thecode.media/jquery/",
"https://thecode.media/split-screen/",
"https://thecode.media/calculus/",
"https://thecode.media/big-data-sales/",
"https://thecode.media/ambient/",
"https://thecode.media/fatality/",
"https://thecode.media/biggest-loser/",
"https://thecode.media/wifi/",
"https://thecode.media/nosock/",
"https://thecode.media/variables/",
"https://thecode.media/start_python/",
"https://thecode.media/i-gonna-code/",
"https://thecode.media/sigi-est/",
"https://thecode.media/nes-game/",
"https://thecode.media/live-view/",
"https://thecode.media/remote/",
"https://thecode.media/arduino-code/",
"https://thecode.media/horses/",
"https://thecode.media/runinstein/",
"https://thecode.media/wp-template/",
"https://thecode.media/tilda/",
"https://thecode.media/todo/",
"https://thecode.media/telebot/",
"https://thecode.media/summator-2/",
"https://thecode.media/get-rich-coding/",
"https://thecode.media/content-manager/",
"https://thecode.media/vzrosly-stal/",
"https://thecode.media/py-install/",
"https://thecode.media/quantum/",
"https://thecode.media/dns/",
"https://thecode.media/practicum/",
"https://thecode.media/react/",
"https://thecode.media/1september/",
"https://thecode.media/summator/",
"https://thecode.media/vds/",
"https://thecode.media/made-in-china/",
"https://thecode.media/bar/",
"https://thecode.media/zodiac/",
"https://thecode.media/crc32/",
"https://thecode.media/css-links/",
"https://thecode.media/oop_battle/",
"https://thecode.media/be-combo/",
"https://thecode.media/unity/",
"https://thecode.media/data-science/",
"https://thecode.media/junior/",
"https://thecode.media/qc/",
"https://thecode.media/be-middle/",
"https://thecode.media/senior/",
"https://thecode.media/teamlead/",
"https://thecode.media/frontend/",
"https://thecode.media/lift/",
"https://thecode.media/be-fuzzy/",
"https://thecode.media/best-2020/",
"https://thecode.media/git/",
"https://thecode.media/stt-cloud/",
"https://thecode.media/matrix-pills/",
"https://thecode.media/na-stile/",
"https://thecode.media/no-coffee/",
"https://thecode.media/framelibs/",
"https://thecode.media/children/",
"https://thecode.media/balls-possibly/",
"https://thecode.media/le-meduza/",
"https://thecode.media/electricity/",
"https://thecode.media/tailored-swift/",
"https://thecode.media/objective/",
"https://thecode.media/host/",
"https://thecode.media/go-public/",
"https://thecode.media/how-internet-works-1/",
"https://thecode.media/domain/",
"https://thecode.media/this-is-object/",
"https://thecode.media/ole-ole-ole/",
"https://thecode.media/thousand/",
"https://thecode.media/average/",
"https://thecode.media/stt-python/",
"https://thecode.media/ping-pong/",
"https://thecode.media/pygames/",
"https://thecode.media/odobreno/",
"https://thecode.media/qwerty123/",
"https://thecode.media/neurocorrector/",
"https://thecode.media/neuro-cam/",
"https://thecode.media/10-jquery/",
"https://thecode.media/repeat/",
"https://thecode.media/assembler/",
"https://thecode.media/sublime-one-love/",
"https://thecode.media/zloy/",
"https://thecode.media/mariya-ivanovna/",
"https://thecode.media/ruby/",
"https://thecode.media/electron-password/",
"https://thecode.media/plane/",
"https://thecode.media/glitch/",
"https://thecode.media/security/",
"https://thecode.media/stupid-2019/",
"https://thecode.media/jquery-search/",
"https://thecode.media/pimp-my-pass/",
"https://thecode.media/text-ultimate/",
"https://thecode.media/hurry/",
"https://thecode.media/siri/",
"https://thecode.media/zero-cool/",
"https://thecode.media/small-talk/",
"https://thecode.media/die-hard/",
"https://thecode.media/le-piton/",
"https://thecode.media/hr-code/",
"https://thecode.media/nano-code/",
"https://thecode.media/the_question/",
"https://thecode.media/godlike/",
"https://thecode.media/be-logic/",
"https://thecode.media/snake-js/",
"https://thecode.media/be-mobile/",
"https://thecode.media/baboolya/",
"https://thecode.media/timelag/",
"https://thecode.media/doors/",
"https://thecode.media/phone-code/",
"https://thecode.media/snake-arduino/",
"https://thecode.media/css-intro/",
"https://thecode.media/le-timer/",
"https://thecode.media/oop_battle/",
"https://thecode.media/good-morning/",
"https://thecode.media/study-bot/",
"https://thecode.media/python-bot/",
"https://thecode.media/robot-quiz/",
"https://thecode.media/hacking-quiz/",
"https://thecode.media/lulz-quiz/",
"https://thecode.media/hard-quiz/",
"https://thecode.media/torrent/",
"https://thecode.media/travel/",
"https://thecode.media/le-snob/",
"https://thecode.media/no-spagetti/",
"https://thecode.media/house/",
"https://thecode.media/cryptorush/",
"https://thecode.media/coronarelax/",
"https://thecode.media/pure/",
"https://thecode.media/c-cpp/",
"https://thecode.media/machine-loving/",
"https://thecode.media/orwell/",
"https://thecode.media/darknet/",
"https://thecode.media/ai/",
"https://thecode.media/oop-class/",
"https://thecode.media/cookie/",
"https://thecode.media/malware/",
"https://thecode.media/ftp/",
"https://thecode.media/html/",
"https://thecode.media/java/",
"https://thecode.media/php-haters/",
"https://thecode.media/tor/",
"https://thecode.media/crack-safe/",
"https://thecode.media/epidemic/",
"https://thecode.media/hash-brown/",
"https://thecode.media/java-js/",
"https://thecode.media/js-types/",
"https://thecode.media/losers/",
"https://thecode.media/ssl/",
"https://thecode.media/uncaughtsyntaxerror-unexpected-identifier/",
"https://thecode.media/uncaughtsyntaxerror-unexpected-token/",
"https://thecode.media/uncaughttyperrror-cannot-read-property/",
"https://thecode.media/mobile-dev/",
"https://thecode.media/verevka/",
"https://thecode.media/speed/",
"https://thecode.media/buckwheat/",
"https://thecode.media/distance/",
"https://thecode.media/node-js/",
"https://thecode.media/pascal/",
"https://thecode.media/ill-be-clean/",
"https://thecode.media/to-be-back/",
"https://thecode.media/replaceable/",
"https://thecode.media/code-review/",
"https://thecode.media/gasoline/",
"https://thecode.media/to-be-test/",
"https://thecode.media/scala/",
"https://thecode.media/row-power/",
"https://thecode.media/percent/",
"https://thecode.media/things/",
"https://thecode.media/prof-newsletter/",
"https://thecode.media/backend/",
"https://thecode.media/immortal-pong/",
"https://thecode.media/blind/",
"https://thecode.media/go-faster/",
"https://thecode.media/cpp/",
"https://thecode.media/uncaught-syntaxerror-unexpected-end-of-input/",
"https://thecode.media/stress-quiz/",
"https://thecode.media/secret-pong/",
"https://thecode.media/override/",
"https://thecode.media/whg/",
"https://thecode.media/profit/",
"https://thecode.media/memas/",
"https://thecode.media/digital-sound/",
"https://thecode.media/api/",
"https://thecode.media/be-math-2/",
"https://thecode.media/backup/",
"https://thecode.media/backup-master/",
"https://thecode.media/glvrd/",
"https://thecode.media/id/",
"https://thecode.media/uncaught-syntaxerror-missing-after-argument-list/",
"https://thecode.media/ex-startup/",
"https://thecode.media/doom-everywhere/",
"https://thecode.media/template-one/",
"https://thecode.media/david-roganov/",
"https://thecode.media/spacex/",
"https://thecode.media/webstorm/",
"https://thecode.media/json/",
"https://thecode.media/treger/",
"https://thecode.media/ya-blitz/",
"https://thecode.media/radius/",
"https://thecode.media/xhr/",
"https://thecode.media/treger2/",
"https://thecode.media/raidemption/",
"https://thecode.media/chief-technical-officer/",
"https://thecode.media/summary/",
"https://thecode.media/ex-wallpaper/",
"https://thecode.media/soap/",
"https://thecode.media/decompose/",
"https://thecode.media/desc/",
"https://thecode.media/sprint/",
"https://thecode.media/bye-or-die/",
"https://thecode.media/who-win/",
"https://thecode.media/vladimir-olokhtonov/",
"https://thecode.media/lossless/",
"https://thecode.media/parse/",
"https://thecode.media/typeerror-is-not-an-abject/",
"https://thecode.media/backup-me/",
"https://thecode.media/stress-test/",
"https://thecode.media/syntaxerror-missing-formal-parameter/",
"https://thecode.media/start-fast/",
"https://thecode.media/halkechev/",
"https://thecode.media/halkechev2/",
"https://thecode.media/le-design/",
"https://thecode.media/syntaxerror-missing-after-property-id/",
"https://thecode.media/attrb-mthd/",
"https://thecode.media/headphones/",
"https://thecode.media/active-noise-cancelling/",
"https://thecode.media/remote-work-quiz/",
"https://thecode.media/garbage/",
"https://thecode.media/ubuntu-linux/",
"https://thecode.media/trie/",
"https://thecode.media/func/",
"https://thecode.media/laravel/",
"https://thecode.media/save-json/",
"https://thecode.media/syntaxerror-missing-after-formal-parameters/",
"https://thecode.media/recursion/",
"https://thecode.media/haskell/",
"https://thecode.media/gen/",
"https://thecode.media/db/",
"https://thecode.media/boosting/",
"https://thecode.media/pavel-sviridov/",
"https://thecode.media/mnogo/",
"https://thecode.media/sokr/",
"https://thecode.media/dbsm/",
"https://thecode.media/pik-balmera/",
"https://thecode.media/kanban/",
"https://thecode.media/check-list/",
"https://thecode.media/text-quiz/",
"https://thecode.media/mysql/",
"https://thecode.media/mysql/",
"https://thecode.media/rust/",
"https://thecode.media/manage-this/",
"https://thecode.media/altshuller/",
"https://thecode.media/interview/",
"https://thecode.media/fotorama/",
"https://thecode.media/tetris/",
"https://thecode.media/ai-tetris/",
"https://thecode.media/scrum/",
"https://thecode.media/speed-two/",
"https://thecode.media/quick-share/",
"https://thecode.media/stack/",
"https://thecode.media/mobile-first/",
"https://thecode.media/nosql/",
"https://thecode.media/narazves/",
"https://thecode.media/oop-class-2/",
"https://thecode.media/design-first/",
"https://thecode.media/arcanoid/",
"https://thecode.media/donut/",
"https://thecode.media/casino/",
"https://thecode.media/heap/",
"https://thecode.media/rust/",
"https://thecode.media/float/",
"https://thecode.media/markdown/",
"https://thecode.media/books/",
"https://thecode.media/daniil-popov/",
"https://thecode.media/android-developer/",
"https://thecode.media/symbols/",
"https://thecode.media/oauth/",
"https://thecode.media/kotlin/",
"https://thecode.media/todo/",
"https://thecode.media/plotly/",
"https://thecode.media/no-digit-code/",
"https://thecode.media/asymmetric/",
"https://thecode.media/qi/",
"https://thecode.media/vernam/",
"https://thecode.media/vernam-js/",
"https://thecode.media/shtykov/",
"https://thecode.media/memory/",
"https://thecode.media/ark/",
"https://thecode.media/7-oshibok-na-sobesedovanii/",
"https://thecode.media/dh/",
"https://thecode.media/typescript/",
"https://thecode.media/stark/",
"https://thecode.media/crypto/",
"https://thecode.media/zapusk-2/",
"https://thecode.media/fingerprint/",
"https://thecode.media/puzzle/",
"https://thecode.media/python-time-2/",
"https://thecode.media/no-chance/",
"https://thecode.media/lossy/",
"https://thecode.media/1wire/",
"https://thecode.media/pasha-flipper/",
"https://thecode.media/perl/",
"https://thecode.media/alexey-vasilev/",
"https://thecode.media/viasat/",
"https://thecode.media/podcast/",
"https://thecode.media/copy-ya-ru/",
"https://thecode.media/mircrosd/",
"https://thecode.media/bash/",
"https://thecode.media/rotation/",
"https://thecode.media/css-grid/",
"https://thecode.media/train/",
"https://thecode.media/grid-2/",
"https://thecode.media/za-proezd/",
"https://thecode.media/grid-3/",
"https://thecode.media/david/",
"https://thecode.media/alien-vs-predator/",
"https://thecode.media/grid-portfolio/",
"https://thecode.media/podcast-lavka/",
"https://thecode.media/it-start-2/",
"https://thecode.media/anastasiya-nikulina/",
"https://thecode.media/linter/",
"https://thecode.media/bomberman/",
"https://thecode.media/5-linters/",
"https://thecode.media/lineynaya-algebra-vektory/",
"https://thecode.media/no-nda/",
"https://thecode.media/code-swap/",
"https://thecode.media/how-to-start/",
"https://thecode.media/referenceerror-invalid-left-hand-side-in-assignment/",
"https://thecode.media/vectors-operations/",
"https://thecode.media/oop-abstract/",
"https://thecode.media/leonov/",
"https://thecode.media/lucky-strike/",
"https://thecode.media/browser/",
"https://thecode.media/2020/",
"https://thecode.media/3d-stars/",
"https://thecode.media/anna-leonova/",
"https://thecode.media/cold-fusion/",
"https://thecode.media/normalize/",
"https://thecode.media/hotkey/",
"https://thecode.media/oven/",
"https://thecode.media/vim/",
"https://thecode.media/draw/",
"https://thecode.media/visual-studio-code/",
"https://thecode.media/tetris-2/",
"https://thecode.media/start-now/",
"https://thecode.media/lapsha-1/",
"https://thecode.media/cubism/",
"https://thecode.media/lapsha-2/",
"https://thecode.media/zerocode/",
"https://thecode.media/cat/",
"https://thecode.media/static/",
"https://thecode.media/komm/",
"https://thecode.media/path-js/",
"https://thecode.media/cloudly/",
"https://thecode.media/haters-gonna-code-2/",
"https://thecode.media/csp/",
"https://thecode.media/mitin-says-no/",
"https://thecode.media/hire-js/",
"https://thecode.media/tableau/",
"https://thecode.media/impossible/",
"https://thecode.media/csp-on/",
"https://thecode.media/maze/",
"https://thecode.media/mix/",
"https://thecode.media/le-beton/",
"https://thecode.media/3d-print/",
"https://thecode.media/5-and-a-half/",
"https://thecode.media/lineynaya-zavisimost-vektorov/",
"https://thecode.media/fast-m1/",
"https://thecode.media/ninja-run/",
"https://thecode.media/matrix-101/",
"https://thecode.media/arm-x86/",
"https://thecode.media/piano-js/",
"https://thecode.media/10-swift/",
"https://thecode.media/travel-plane/",
"https://thecode.media/extention/",
"https://thecode.media/obratnaya-matritsa/",
"https://thecode.media/svg/",
"https://thecode.media/freelance/",
"https://thecode.media/brat-2/",
"https://thecode.media/angular/",
"https://thecode.media/rgb/",
"https://thecode.media/10-go/",
"https://thecode.media/coffee/",
]

# открываем текстовый файл, куда будем добавлять заголовки
file = open("zag.txt", "a")

# перебираем все адреса из списка
for x in url:
    # получаем исходный код очередной страницы из списка
    html_code = str(urlopen(x).read(),'utf-8')
    # отправляем исходный код страницы на обработку в библиотеку
    soup = BeautifulSoup(html_code, "html.parser")

    # находим название страницы с помощью метода find()
    s = soup.find('title').text

    # выводим его на экран
    print(s)

    # сохраняем заголовок в файле и переносим курсор на новую строку
    file.write(s + '. ')

# закрываем файл
file.close()

Что дальше

А даль­ше логич­ное про­дол­же­ние — про­грам­ма на цепях Мар­ко­ва, кото­рая будет гене­ри­ро­вать заго­лов­ки для ста­тей Кода на осно­ве наших ста­рых заголовков.

Текст:

Миха­ил Полянин

Редак­ту­ра:

Мак­сим Ильяхов

Худож­ник:

Даня Бер­ков­ский

Кор­рек­тор:

Ири­на Михеева

Вёрст­ка:

Мария Дро­но­ва

Соц­се­ти:

Олег Веш­кур­цев