Home Forums Main Forums Python Forum Ideas needed! How to develop an automatic web update alert by Python?

  • Ideas needed! How to develop an automatic web update alert by Python?

     Datura updated 3 years, 11 months ago 2 Members · 6 Posts
  • Datura

    Member
    October 26, 2020 at 10:48 am
    Up
    0
    Down

    A friend needs an automatic web alert for her business. For example, if there are any new updates or modifications at out Activity page:

    Activity Feed

    Then she can receive an alert of email notification automatically.

    My rough idea is:

    1. we can develop a web crawler by Python and get all the current web contents and find out the latest update date, modification date or activity date. Create a data set for the current contents.

    2. Use Python scheduler to run the crawler each day or each hour to get the new web contents. Then we can compare the new latest update/modification dates etc with those from our history data.

    3. If the new dates are later than historical ones, it means that the website is updated or modified. We can then send out an email alert and append the newly grabbed data into history data. If not, we do nothing but keep on checking the website by our crawler.

    Is there any better method than this one? Please let me know, I am all ears, thank you!

    Patrick, how do you think? You are an expert, please give me your suggestions. Thanks!

  • Patrick

    Member
    October 26, 2020 at 10:49 am
    Up
    0
    Down

    My 2 cents:

    1. If it is possible use “push” instead of “pull”, for example push updates/alerts to emails with accurate changes;

    2. If push is not available, try free webpage monitoring servies for example: https://visualping.io/

    3. There are also desktop version of monitoring tools

    4. Customize using Python tools: for example https://thp.io/2008/urlwatch/

    5. Write your own code as what you’ve mentioned above

    • Datura

      Member
      October 26, 2020 at 10:59 am
      Up
      0
      Down

      Thank you for your quick reply and valuable suggestions.

      “1. If it is possible use “push” instead of “pull”, for example push updates/alerts to emails with accurate changes; “

      I am not totally clear. You mean forward the new changes by email? It is a good idea but requires more work.

      Let me check the webpage monitoring services available now, the Method 4 by Python tools is probably the best option because we can customize it.

      • Patrick

        Member
        October 26, 2020 at 11:07 am
        Up
        0
        Down

        I meant that if your friend’s website is owned by himself/herself, then the changes are already known in the database. It is easier to do a lot of alert either by scheduler, by db table trigger, or other mechanism without consistently checking on this webpage every minute or so with extra web traffic without knowing what exactly the change types are (forum/posts/rely, etc.)

        If the website you are monitoring is owned by other people then the 1st option doesn’t really exist.

        I am not totally clear. You mean forward the new changes by email? It is a good idea but requires more work.

        • Datura

          Member
          October 26, 2020 at 11:10 am
          Up
          0
          Down

          Got it. She wants to monitor government websites, they definitely don’t belong to her~~ 🙂

    • Datura

      Member
      October 26, 2020 at 11:13 am
      Up
      0
      Down

      I checked the urlwatch Python package, please see below, the fundamental idea is same as mine. Please see below…… we can just use this package since it is available and free.

      ———————————————-

      Introduction

      urlwatch monitors the output of webpages or arbitrary shell commands.

      Every time you run urlwatch, it:

      • retrieves the output and processes it
      • compares it with the version retrieved the previous time (“diffing”)
      • if it finds any differences, generates a summary “report” that can be displayed or sent via one or more methods, such as email

      • This reply was modified 3 years, 11 months ago by  Datura.
      • This reply was modified 3 years, 11 months ago by  Datura.

Log in to reply.

Original Post
0 of 0 posts June 2018
Now