How to create multiple data frames from a list?

[Datura]

Up

0

Down

When using web crawler to extract ranking data for immigration from government websites, the extracted data are stored in a list of 20 dataframe elements.

!pip install requests
!pip install bs4
!pip install lxml
import requests        
from bs4 import BeautifulSoup

url='https://www.canada.ca/en/immigration-refugees-citizenship/services/immigrate-canada/express-entry/eligibility/criteria-comprehensive-ranking-system/grid.html'
html = requests.get(url).content
raw = pd.read_html(html) 
type(raw)       #raw is a list
len(raw)        # raw has 20 elements

Then, we want to create 20 data frames for the 20 list elements. We can do it manually as below:

A_Core = raw[0]
B_Spouse=raw[1]
C_Transfer_Education=raw[2]
C_Transfer_Foreign=raw[3]
C_Transfer_Certificate=raw[4]
D_Additional=raw[5]
Age=raw[6]

This repetitive work is really annoying because we have so many data frames to create. Can we use looping to create the 20 data frames automatically???

Who can develop the code? Please share with us, thanks!

• This discussion was modified 3 years, 6 months ago by  Datura.
• This discussion was modified 3 years, 6 months ago by  Datura.
• This discussion was modified 3 years, 6 months ago by  Justin.

[yiyiyi]

Up

0

Down

Hello, very good question, but in this case, we don’t need to convert these 20 elements into date frames because they are already date frames. You could use isinstance(raw[X], pd.DataFrame) to check and we will see that these are data frames already.

[Datura]

Up

0

Down

No. All these tables are totally different with different columns and structures, I need separate them and then use each one differently.

[yiyiyi]

Up

0

Down

And then to answer your loop question, if you’d like to read in these df using loops, this is definitely feasible. We could simply do something like this: for i in range(Len(row)): [list of names you defined before] [i] = raw[i]

[Datura]

Up

0

Down

I tried this method, it run through without errors, but all the data frames df1-df3 are still all empty. The loop does not overwrite the pre-defined empty ones.

df1=pd.DataFrame()
df2=pd.DataFrame()
df3=pd.DataFrame()

dflist=[df1,df2,df3]
for i in range(len(dflist)): 
    print("loop round: ", i)
    temp=raw[i]
    print(temp)
    dflist[i] = temp

df1

So, what’s wrong?

• This reply was modified 3 years, 6 months ago by  Datura.

[yiyiyi]

Up

1

Down

because when we call every element from the list, the one we are calling is actually the name. So what I refers is:

we need the name but not overwrite. You could try:

locals()[“df”+ str(i) ] = raw[i]

this will create 20 dfs with names df1 … df20 like you did

• This reply was modified 3 years, 6 months ago by  Justin.
• This reply was modified 3 years, 6 months ago by  Justin.

[Justin]

Up

0

Down

Yes, I think this is probably the right way, let me try……. can you talk about the macro functions: locals() and global() in Python? You can use open a new discussion. Thanks!

[Datura]

Up

1

Down

Wow, it works, awesome! you are a genius. Thank you! We need to use the locals() /globals() macro symbol tables, which are similar to those in SAS.

df1=pd.DataFrame({})
df2=pd.DataFrame({})
df3=pd.DataFrame({})
df4=pd.DataFrame({})
df5=pd.DataFrame({})
df6=pd.DataFrame({})
dflist=[df1,df2,df3, df4, df5, df6]
for i in range(len(dflist)): 
    print("loop round: ", i)
    locals()['df'+str(i+1)] = raw[i]
        
df6

• This reply was modified 3 years, 6 months ago by  Datura.

[Patrick]

Up

1

Down

Content removed.

• This reply was modified 3 years, 6 months ago by  Patrick. Reason: irrelevant

[Datura]

Up

0

Down

??? Is this the method to extract table data without using the read_html() function?

[Anonymous]

Up

0

Down

if you need the dataframes created, the raw has already a list of dataframes. You can refer to it by raw[0], ….. etc. If you want to df1, df2, … instead of df[1], df[2] style, you have to do it manually.

[Datura]

Up

0

Down

No automatic method? OK, it’s too bad, really inconvenient to use raw[1], raw[2]……

• This reply was modified 3 years, 6 months ago by  Datura.

[Justin]

Up

0

Down

Thank all for the contribution and discussion, very useful and helpful.

Based on Yi’s contribution, we can go one step further. If the newly created data frames don’t have any similar pattern, we can use below method to create all of them.

### create a list of dataframe names.
dflist=['Core', 'Spouse', 'Education', 'Work', 'Certificate', 'Additional']  

### looping over each element
for i in range(len(dflist)): 
    print("loop round: ", i)
    globals()[dflist[i]] = raw[i]
    
print(Work)

It will create the 6 data frames: Core, Spouse… Additional. Ideas on ideas, this is a more generic way to do it.