Home › Forums › Main Forums › Python Forum › How to create multiple data frames from a list?
-
How to create multiple data frames from a list?
-
When using web crawler to extract ranking data for immigration from government websites, the extracted data are stored in a list of 20 dataframe elements.
!pip install requests
!pip install bs4
!pip install lxml
import requests
from bs4 import BeautifulSoup
url='https://www.canada.ca/en/immigration-refugees-citizenship/services/immigrate-canada/express-entry/eligibility/criteria-comprehensive-ranking-system/grid.html'
html = requests.get(url).content
raw = pd.read_html(html)
type(raw) #raw is a list
len(raw) # raw has 20 elementsThen, we want to create 20 data frames for the 20 list elements. We can do it manually as below:
A_Core = raw[0]
B_Spouse=raw[1]
C_Transfer_Education=raw[2]
C_Transfer_Foreign=raw[3]
C_Transfer_Certificate=raw[4]
D_Additional=raw[5]
Age=raw[6]This repetitive work is really annoying because we have so many data frames to create. Can we use looping to create the 20 data frames automatically???
Who can develop the code? Please share with us, thanks!
-
-
-
I tried this method, it run through without errors, but all the data frames df1-df3 are still all empty. The loop does not overwrite the pre-defined empty ones.
df1=pd.DataFrame()
df2=pd.DataFrame()
df3=pd.DataFrame()
dflist=[df1,df2,df3]
for i in range(len(dflist)):
print("loop round: ", i)
temp=raw[i]
print(temp)
dflist[i] = temp
df1So, what’s wrong?
- This reply was modified 3 years, 11 months ago by Datura.
-
because when we call every element from the list, the one we are calling is actually the name. So what I refers is:
we need the name but not overwrite. You could try:
locals()[“df”+ str(i) ] = raw[i]
this will create 20 dfs with names df1 … df20 like you did
-
Wow, it works, awesome! you are a genius. Thank you! We need to use the locals() /globals() macro symbol tables, which are similar to those in SAS.
df1=pd.DataFrame({})
df2=pd.DataFrame({})
df3=pd.DataFrame({})
df4=pd.DataFrame({})
df5=pd.DataFrame({})
df6=pd.DataFrame({})dflist=[df1,df2,df3, df4, df5, df6]
for i in range(len(dflist)):
print("loop round: ", i)
locals()['df'+str(i+1)] = raw[i]
df6- This reply was modified 3 years, 11 months ago by Datura.
-
-
-
Anonymous
Deleted UserOctober 26, 2020 at 4:49 pm -
Thank all for the contribution and discussion, very useful and helpful.
Based on Yi’s contribution, we can go one step further. If the newly created data frames don’t have any similar pattern, we can use below method to create all of them.
### create a list of dataframe names.
dflist=['Core', 'Spouse', 'Education', 'Work', 'Certificate', 'Additional']
### looping over each element
for i in range(len(dflist)):
print("loop round: ", i)
globals()[dflist[i]] = raw[i]
print(Work)It will create the 6 data frames: Core, Spouse… Additional. Ideas on ideas, this is a more generic way to do it.
Log in to reply.