I am trying to get all the results it shows in the shell to go into my text file but it will only put the last one found. I’m sure its a simple but I have tried several ways and can’t seem to figure out how to get all the results into my text file!
res = requests.get('https://www.infosecblog.org//').text
soup = bs4.BeautifulSoup(res , 'lxml')
writer = ('headline', 'summary', 'url')
for article in soup.find_all('article'):
headline = article.h1.a.text
print(headline)
summary = article.find_all('div', class_='entry-content')
for s in summary:
print(summary)
# Grabbing an anchor tag using dictionary formating
url = article.find('a')['href']
print(url)
print()
with open('scrape.txt', 'w') as file:
csv_writer = csv.writer(file)
csv_writer.writerow([headline])
csv_writer.writerow([summary])
csv_writer.writerow([url])
Hi, it seems you’re using Python, try to change the open function mode to “append” by changing the line with open('scrape.txt', 'w') as file:
to with open('scrape.txt', 'a') as file:
Yes, it is python I fixed my topic header so others will know. Unfortunately, I have tried this and get the same result. I think I may have to setup a larger loop to encapsulate it so that it will continue to run and hopefully write the data every time it runs.
It still works but not for the writer portion its giving me this error: Traceback (most recent call last):
File “C:\Users\ajohn\Documents\scrapping.py”, line 46, in
csv_writer.writerow([summary])
File “C:\Users\ajohn\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py”, line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: ‘charmap’ codec can’t encode character ‘\U0001f609’ in position 659: character maps to