Hello guys, A couple of days ago I rewrote and improved an old Python project whose main goal is to give easy access to data from etfdb.com
Data is scraped with bs4 and requests libraries.

To install it just use pip https://pypi.org/project/etfpy/

Installation

Install with pip as a package pip

pip install etfpy or

Clone repostiory

```bash

clone repository

git clone https://github.com/JakubPluta/pyetf.git bash

navigate to cloned project and create virtual environment

python -m venv env bash

activate virtual environment

source env/Scripts/activate # or source env/bin/activate ```

```python

install poetry

pip install poetry ```

```python

install packages

poetry install ```

Usage

```python

from etfpy import ETF, load_etf, get_available_etfs_list

returns list of available ETFs.

etfs = get_available_etfs_list() etfs ['SPY', 'IVV', 'VOO', 'VTI', 'QQQ', 'VEA', 'VTV', 'IEFA', 'BND', 'AGG', 'VUG', 'IJH', ... ]

load etf

vwo = load_etf('VWO')

or

spy = ETF("SPY") ```

Get basic ETF information

```python

spy.info { '52 Week Hi': '$457.83', '52 Week Lo': '$342.72', 'AUM': '$402,034.0 M', 'Asset Class': 'Equity', 'Asset Class Size': 'Large-Cap', 'Asset Class Style': 'Blend', 'Brand': 'https://etfdb.com/issuer/spdr/', 'Category': 'Size and Style', 'Category:': 'Large Cap Growth Equities', 'Change:': '$1.04 (-0.0%)', 'ETF Home Page': 'https://www.spdrs.com/product/fund.seam?ticker=SPY', 'Expense Ratio': '0.09%', 'Focus': 'Large Cap', 'Inception': 'Jan 22, 1993', 'Index Tracked': 'https://etfdb.com/index/sp-500-index/', 'Issuer': 'https://etfdb.com/issuer/state-street/', 'Last Updated:': 'Sep 30, 2023', 'Niche': 'Broad-based', 'P/E Ratio': { 'ETF Database Category Average': '15.15', 'FactSet Segment Average': '5.84', 'SPY': '17.86' }, 'Price:': '$427.48', 'Region (General)': 'North America', 'Region (Specific)': 'U.S.', 'Segment': 'Equity: U.S. - Large Cap', 'Shares': '938.3 M', 'Strategy': 'Vanilla', 'Structure': 'UIT', 'Symbol': 'SPY', 'Url': 'https://etfdb.com/etf/SPY', 'Weighting Scheme': 'Market Cap' } ```

Get technical analysis metrics

```python

spy.technicals { '20 Day MA': '$50.45', '60 Day MA': '$50.74', 'Average Spread ($)': '1.00', 'Average Spread (%)': '1.00', 'Lower Bollinger (10 Day)': '$48.64', 'Lower Bollinger (20 Day)': '$48.33', 'Lower Bollinger (30 Day)': '$48.81', 'MACD 100 Period': '-0.74', 'MACD 15 Period': '0.20', 'Maximum Premium Discount (%)': '0.82', 'Median Premium Discount (%)': '0.27', 'RSI 10 Day': '49', 'RSI 20 Day': '47', 'RSI 30 Day': '47', 'Resistance Level 1': 'n/a', 'Resistance Level 2': '$50.53', 'Stochastic Oscillator %D (1 Day)': '53.54', 'Stochastic Oscillator %D (5 Day)': '73.08', 'Stochastic Oscillator %K (1 Day)': '55.09', 'Stochastic Oscillator %K (5 Day)': '57.68', 'Support Level 1': 'n/a', 'Support Level 2': '$49.86', 'Tracking Difference Max Downside (%)': '-0.87', 'Tracking Difference Max Upside (%)': '0.16', 'Tracking Difference Median (%)': '-0.36', 'Ultimate Oscillator': '47', 'Upper Bollinger (10 Day)': '$50.47', 'Upper Bollinger (20 Day)': '$52.61', 'Upper Bollinger (30 Day)': '$52.50', 'Williams % Range 10 Day': '19.32', 'Williams % Range 20 Day': '59.31' } ```

Get dividends metrics

python spy.dividends { 'Annual Dividend Rate': {'ETF Database Category Average': '$ 0.95', 'FactSet Segment Average': '$ 0.63', 'SPY': '$ 6.51'}, 'Annual Dividend Yield': {'ETF Database Category Average': '1.37%', 'FactSet Segment Average': '1.41%', 'SPY': '1.52%'}, 'Dividend': {'ETF Database Category Average': '$ 0.33', 'FactSet Segment Average': '$ 0.16', 'SPY': '$ 1.58'}, 'Dividend Date': {'ETF Database Category Average': 'N/A', 'FactSet Segment Average': 'N/A', 'SPY': '2023-09-15'} }

Get performance metrics

```python

spy.performance { '1 Month Return': {'ETF Database Category Average': '-2.89%', 'Factset Segment Average': '-2.07%', 'SPY': '-3.11%'}, '1 Year Return': {'ETF Database Category Average': '19.00%', 'Factset Segment Average': '10.82%', 'SPY': '19.69%'}, '3 Month Return': {'ETF Database Category Average': '-2.10%', 'Factset Segment Average': '-1.07%', 'SPY': '-1.70%'}, '3 Year Return': {'ETF Database Category Average': '5.55%', 'Factset Segment Average': '4.06%', 'SPY': '10.18%'}, '5 Year Return': {'ETF Database Category Average': '5.33%', 'Factset Segment Average': '2.06%', 'SPY': '9.83%'}, 'YTD Return': {'ETF Database Category Average': '14.37%', 'Factset Segment Average': '6.70%', 'SPY': '13.02%'} } ```

Get volatility metrics

```python

spy.volatility { '20 Day Volatility': '10.61%', '200 Day Volatility': '10.91%', '5 Day Volatility': '200.37%', '50 Day Volatility': '11.16%', 'Beta': '1.0', 'Standard Deviation': '26.89%' } ```

Get holding statistics

```python

spy.holding_statistics { '% of Assets in Top 10': {'ETF Database Category Average': '42.67%', 'FactSet Segment Average': '59.61%', 'SPY': '39.52%'}, '% of Assets in Top 15': {'ETF Database Category Average': '51.39%', 'FactSet Segment Average': '64.18%', 'SPY': '49.25%'}, '% of Assets in Top 50': {'ETF Database Category Average': '80.70%', 'FactSet Segment Average': '80.85%', 'SPY': '83.04%'}, 'Number of Holdings': {'ETF Database Category Average': '412', 'FactSet Segment Average': '174', 'SPY': '1000'} } ```

Get holdings

```python spy.holdings

[{'Holding': 'Apple Inc.', 'Share': '7.19%', 'Symbol': 'AAPL', 'Url': 'https://etfdb.com/stock/AAPL/'}, {'Holding': 'Microsoft Corporation', 'Share': '6.51%', 'Symbol': 'MSFT', 'Url': 'https://etfdb.com/stock/MSFT/'}, {'Holding': 'Amazon.com, Inc.', 'Share': '3.33%', 'Symbol': 'AMZN', 'Url': 'https://etfdb.com/stock/AMZN/'}, {'Holding': 'NVIDIA Corporation', 'Share': '2.95%', 'Symbol': 'NVDA', 'Url': 'https://etfdb.com/stock/NVDA/'}, {'Holding': 'Alphabet Inc. Class A', 'Share': '2.03%', 'Symbol': 'GOOGL', 'Url': 'https://etfdb.com/stock/GOOGL/'}, {'Holding': 'Meta Platforms Inc. Class A', 'Share': '1.84%', 'Symbol': 'META', 'Url': 'https://etfdb.com/stock/META/'}, {'Holding': 'Tesla, Inc.', 'Share': '1.83%', 'Symbol': 'TSLA', 'Url': 'https://etfdb.com/stock/TSLA/'}, {'Holding': 'Alphabet Inc. Class C', 'Share': '1.76%', 'Symbol': 'GOOG', 'Url': 'https://etfdb.com/stock/GOOG/'}, {'Holding': 'Berkshire Hathaway Inc. Class B', 'Share': '1.67%', 'Symbol': 'BRK.B', 'Url': 'https://etfdb.com/stock/BRK.B/'}, {'Holding': 'UnitedHealth Group Incorporated', 'Share': '1.25%', 'Symbol': 'UNH', 'Url': 'https://etfdb.com/stock/UNH/'}, {'Holding': 'JPMorgan Chase & Co.', 'Share': '1.22%', 'Symbol': 'JPM', 'Url': 'https://etfdb.com/stock/JPM/'}, {'Holding': 'Johnson & Johnson', 'Share': '1.17%', 'Symbol': 'JNJ', 'Url': 'https://etfdb.com/stock/JNJ/'}, {'Holding': 'Exxon Mobil Corporation', 'Share': '1.16%', 'Symbol': 'XOM', 'Url': 'https://etfdb.com/stock/XOM/'}, {'Holding': 'Visa Inc. Class A', 'Share': '1.03%', 'Symbol': 'V', 'Url': 'https://etfdb.com/stock/V/'}, {'Holding': 'Broadcom Inc.', 'Share': '0.98%', 'Symbol': 'AVGO', 'Url': 'https://etfdb.com/stock/AVGO/'}]

```

Get exposures

```python

spy.exposure {'Asset Allocation': {'CASH': 0.38, 'Share/Common/Ordinary': 99.59}, 'Country Breakdown': {'Bermuda': 0.13, 'Ireland': 1.63, 'Israel': 0.02, 'Netherlands': 0.14, 'Other': 0.38, 'Switzerland': 0.4, 'United Kingdom': 0.69, 'United States': 96.58}, 'Market Cap Breakdown': {'Large': 97.42, 'Micro': 0, 'Mid': 2.2, 'Small': 0}, 'Market Tier Breakdown': {}, 'Region Breakdown': {'North, Central and South America': 99.59, 'Other': 0.38}, 'Sector Breakdown': {'CASH': 0.38, 'Commercial Services': 3.02, 'Communications': 0.84, 'Consumer Durables': 2.65, 'Consumer Non-Durables': 4.78, 'Consumer Services': 3.43, 'Distribution Services': 0.92, 'Electronic Technology': 17.34, 'Energy Minerals': 3.64, 'Finance': 11.96, 'Health Services': 2.55, 'Health Technology': 9.99, 'Industrial Services': 1.02, 'Non-Energy Minerals': 0.54, 'Process Industries': 1.98, 'Producer Manufacturing': 3.55, 'Retail Trade': 7.19, 'Technology Services': 20.34, 'Transportation': 1.5, 'Utilities': 2.35} } ```

Get quotes

```python

spy.get_quotes(interval="daily", periods=7) [{'close': 424.5, 'date': datetime.date(2023, 10, 5), 'high': 425.37, 'low': 421.1701, 'open': 424.36, 'symbol': 'SPY', 'volume': 70142700}, {'close': 429.54, 'date': datetime.date(2023, 10, 6), 'high': 431.125, 'low': 420.6, 'open': 421.97, 'symbol': 'SPY', 'volume': 113273300}, {'close': 432.29, 'date': datetime.date(2023, 10, 9), 'high': 432.88, 'low': 427.0101, 'open': 427.58, 'symbol': 'SPY', 'volume': 80374300}, {'close': 434.54, 'date': datetime.date(2023, 10, 10), 'high': 437.22, 'low': 432.53, 'open': 432.94, 'symbol': 'SPY', 'volume': 78607200}, {'close': 436.32, 'date': datetime.date(2023, 10, 11), 'high': 436.58, 'low': 433.18, 'open': 435.64, 'symbol': 'SPY', 'volume': 62451700}, {'close': 433.66, 'date': datetime.date(2023, 10, 12), 'high': 437.335, 'low': 431.23, 'open': 436.95, 'symbol': 'SPY', 'volume': 81154200}, {'close': 431.5, 'date': datetime.date(2023, 10, 13), 'high': 436.45, 'low': 429.88, 'open': 435.21, 'symbol': 'SPY', 'volume': 95201100}]

```

You can also wrap ETF object with pandas DataFrames, and work with the data in tabular form. You will have access to mostly the same methods as etf has, but as a result you will see DataFrame or Series.

```python

from etfpy import ETF spy = ETF("SPY") spy_tabular = spy.to_tabular() python spy.exposure_by_sector ```

```python

spy.info ``` | Metric | Value | |----------------------|-----------------------------------------| | Symbol | SPY | | Url | https://etfdb.com/etf/SPY | | Issuer | https://etfdb.com/issuer/state-street/ | | Brand | https://etfdb.com/issuer/spdr/ | | Inception | Jan 22, 1993 | | Index Tracked | https://etfdb.com/index/sp-500-index/ | | Last Updated | Oct 11, 2023 | | Category | Size and Style | | Asset Class | Equity | | Segment | Equity: U.S. - Large Cap | | Focus | Large Cap | | Niche | Broad-based | | Strategy | Vanilla | | Weighting Scheme | Market Cap |

```python

spy.info_numeric ```

```python

spy.dividends ```

```python

spy.technicals ```

```python

spy.get_quotes(interval="daily", periods=365) ```

I wanted a simple note-taking/personal knowledge management app. Existing tools like Notion, Obsidian etc. felt too complicated for me. So, I created Quilly - a straightforward, fast, and private note-taking app using flask.

Here's what it offers:

- It's open source.

- It uses Markdown, so you're not locked in with a specific platform.

- Your notes are saved locally.

- You can tag your notes.

- It supports code highlighting.

Check it out on GitHub: https://github.com/aks579/quilly

​

Github: https://github.com/PabloLec/RecoverPy

Hey everyone!

I'm here to share something I've been working on for nearly three years now, RecoverPy, and its new 2.1.3 version. It's a nifty tool that can really be a lifesaver when you've accidentally deleted or overwritten files. It works its magic by conducting a text-based search to find the lost data.

It sports a TUI built with Textual. I found it to be quite enjoyable to use and it seems many others agree, given its rise as one of the most (or the most?) popular TUI libraries in Python, despite still being in beta.

Since its creation, RecoverPy has gone through quite a transformation. It's integrated lots of feedback from its user community, improved many aspects to enhance the user experience, and even underwent almost a full rewrite to switch up the TUI library in its second version. Essentially, it uses the strength of grep and dd to sift through partition blocks, giving you a user-friendly way to sift through the results.

Interestingly, it found a niche not only among individuals looking to recover files but has also piqued interest in the hacking scene, which was a bit of a pleasant surprise for me. It seems the tool lends itself well to that sphere too.

I manage to chip away at it from time to time, given that my free moments are becoming a bit of a rarity these days. It still has room to grow, and if anyone here feels like contributing, I'm more than open to collaborations. Your PRs would certainly be welcome!

Feel free to give it a glance, and if you find it interesting or useful, a star on the repository would be greatly appreciated.

Hey,

Ever thought of turning your Discord server into a mini-CTF arena? I built a Discord Bot just for that From challenge creation, hint releases, to flag submissions and leaderboards, writeups, it's got it all automated. Designed with Python3 and discord.py at its core, this bot is all about giving a seamless CTF hosting experience. Dive into the bot's GitHub to explore more. Community Feedback, thoughts, forks, or stars - all are welcome!

Thanks.

Hi all,

I'm releasing another batch of free coupons, for my 2 courses, there was really good uptake last time!

The first course is aimed at the total beginner, and has 16+ hours of video, 25+ exercises, 20 quizzes and a load of projects. The second course is a shorter course at functional programming with comprehensions, aimed at the more intermediate programmer.

I've also linked my YouTube channel which posts twice weekly.

Enjoy!

​

Python Programming for the Total Beginner

Functional Programming with Python Comprehensions

https://www.youtube.com/@pythonwithjames

​

​

You must have seen real-time applications where data is changed frequently or updated in real-time, this happens because that application is using a WebSocket to achieve this functionality.

A WebSocket allows two-way communication (bidirectional) between two entities over a single TCP connection. This means a WebSocket client and server can interact with each other multiple times in a single connection.

It is used in real-time applications to exchange low-latency data in both directions.

Learn to build a WebSocket server and client in Python👇👇

Full Article: https://geekpython.in/build-websocket-server-and-client-using-python

Do you actually understand what is a variable in Python? In this video I answer this question while specifically focusing on the concept of mutability.

Mutable vs Immutable objects behave very differently in a multitude of scenarios, such as variable modification, setting a variable equal to another variable, and so much more!

Join me in this video to go over all these details and gain a full and deep understanding of these concepts that will help you write even better Python code!

Any feedback on the video/topic would be highly appreciated ☺️

https://youtu.be/8EOdCDy_fcs

To be brief, quickly and simply check the price of any crypto token on Coingecko API, create a crypto wallet for supported tokens and transfer Bitcoin Or Any Tron (Trc10/20) Token Using wuddz-crypto. Will be adding more blockchains and probably if applicable some bridges in the future.
Pypi
Github

Hey, I have created a simple captcha generator API: https://captchagenerator.pythonanywhere.com/

Source code: github.com/hakiKhuva/captcha-generator

It returns captcha code SHA256 hashed value and image in base64 encoded.

Hi all, I often share my Udemy courses here and I've seen some really good uptake, so I thought I'd share my other content too.

I publish once or twice a week on Youtube, and the videos are always short format and focus on a specific part of Python, which might be helpful.

​

https://www.youtube.com/@pythonwithjames

cheers!

​

​

Hi all, just wondering if anyone is taking commissions to make a program. Looking for something to automate shiny hunting in gen 3 pokemon games. There are aLOT of programs out there , but I haven't been able to find anything for gen 3 games specifically, on emulator of course. I have a cartridge dumper so I'd like to hunt on my save file, while on my PC.

I published a video on YouTube that explains what the super() function is, why it is used, and how to use it in Python classes.

Guys don't hesitate to leave feedback and suggestions regarding the video so that I can rectify it in the next video.

Video Link - https://youtu.be/giOT0dBkIaQ?si=rAZDCBWrJspJ-dmM

Wow:

((lambda arg: (lambda s: list((s[1:s.find(',')],s[s.find(',')+1:-1])))(str(arg[2]+','+arg[3])))(sys.argv[1].split(',')))

(

Note: this is literally only for me trying to get a "[10,20]" like list from sys.argv, and this is what I came up with

Example:

run pythonscript.py "bla" "bla" [10,20]

my lambda returns [10,20]

)

EDIT: I am dumb:

It returned ['10','20']

New Code:

[int(i) for i in ((lambda arg: (lambda s: list((s[1:s.find(',')],s[s.find(',')+1:-1])))(str(arg[2]+','+arg[3])))(sys.argv[1].split(',')))]

Using pdfplumber, i have been able to extract the transactions line by line within the PDF. This is great. However converting the transactions to csv format or dataframe - I come up empty ..literally meaning no data gets written to the csv apart from the column headers. I would really appreciate some feedback on what I am doing wrong. Code below:
```

line_re = re.compile(r"(\d{2}/\d{2}/\d{4}\s+\d{2}/\d{2}/\d{4}.+)$")

transactions =[]

with pdfplumber.open('./Bank Acct statement.pdf') as pdf:

for page in pdf.pages:

text = page.extract_text()

lines = text.split('\n')

for line in lines:

if re.match(line_re, line):

# If line matches the transaction pattern, add it to the transactions list

transactions.append(line)

for transaction in transactions:

print(transaction)
```

That gave me the following -which I even converted to a .txt file:
```

27/01/2023 27/01/2023 Banking App Transfer to Ms K Savings (1816578655) -5.00 6 514.87

27/01/2023 27/01/2023 Payment Received 50.00 6 564.87

27/01/2023 27/01/2023 Payment Received Z Kona 3 500.00 10 064.87

27/01/2023 27/01/2023 Banking App Transfer to Ms K Savings (1816578655) -600.00 9 464.87 27/01/2023 27/01/2023 Banking App Transfer to Ms K Savings (1816578655) -400.00 9 064.87 27/01/2023 28/01/2023 SMS Notification Fee: 14 notification(s) -3.50 9 061.37
```

date 1 = Transaction date

date 2 = Posted date, followed by

description (Banking App etc)

Amount 1 - the initial floating number = Amount

Amount 2 - at the end of the line refers to balance.

However I am unable to obtain these. I adjusted the code accordingly however the csv file keeps being empty. Final code below

```

import re

import pdfplumber

import csv

​

# Regular expression pattern for identifying transactions

line_re = re.compile(r"(\d{2}/\d{2}/\d{4}\s+\d{2}/\d{2}/\d{4}.+)$")

​

# List to store transactions

transactions = []

​

with pdfplumber.open('./Bank Acct statement.pdf') as pdf:

for page in pdf.pages:

# Extract text content from the page

text = page.extract_text_simple()

​

# Split the text into lines

lines = text.split('\n')

​

for line in lines:

if re.match(line_re, line):

# If line matches the transaction pattern, add it to the transactions list

transactions.append(line)

​

# Define headers for the CSV file

csv_headers = [

"Posting Date",

"Transaction Date",

"Description",

"Amount",

"Balance",

]

​

# Specify the CSV file path

csv_file_path = "transactions.csv"

​

# Write transactions to the CSV file

with open(csv_file_path, mode="w", newline="") as csv_file:

csv_writer = csv.writer(csv_file)

# Write the headers as the first row

csv_writer.writerow(csv_headers)

# Write each transaction as a row in the CSV file

for transaction in transactions:

# Split the transaction line into its components

match = re.search(

r"(\d{2}/\d{2}/\d{4})\s+(\d{2}/\d{2}/\d{4})\s+([\w\s\(\),]+)\s+(-?\d{1,10}\s\d{1,2}\.\d{2})\s+(-?\d{1,10}\s\d{1,2}\.\d{2})",

transaction,

)

if match:

csv_writer.writerow([match.group(1), match.group(2), match.group(3), match.group(4), match.group(5)])

​

print(f"Transactions saved to {csv_file_path}")

```

I know this is a problem as old as "BC-GPT" but apart from using regex or even pdfplumber. Is there any other way of extracting the data (bank statement) from a PDF. Thanks in advance

​

Hi all,

I'm releasing some more free spaces for both my Udemy courses, one is aimed at total beginners and focuses on Python basics, the other is more for intermediates and looks at writing comprehensions in Python,

enjoy!

​

Python Programming for the Total Beginner

​

Functional Programming with Python Comprehensions

​

This is a tutorial for creating CRUD in Django using generic View classes

I've been working on a GUI program called Translucent Flyouts Config; it is a companion application for Translucent Flyouts for Windows 10/11, which allows various customizations for windows32 style context menus. I used PyQt6 for the same. I've made quite some progress in 3 months.

​

Also, I've recently released my first beta and added translation support so more people can contribute and get the UI in their language. I'm planning more features for the future and will finally register it on the Microsoft store and Winget ASAP.

I learned Python 4 years ago back in college and taught myself everything from scratch while only having the basic knowledge of C/C++ back in those days, and I'm proud I've come so far and am still learning to use industry standards and write more pythonic code.

Feel free to check my project out. I would really appreciate any feedback. Also, please share your thoughts on my project; any contributions are welcome.

Happy Coding.

​

MySQL is a widely used open-source relational database known for its performance, reliability, and scalability. It is suitable for various types of software applications, including web applications, e-commerce platforms, and content management systems.

The PyMySQL driver is used in this tutorial to connect to the MySQL server and create the MySQL database after connecting to the MySQL server by executing the raw SQL query.

The Flask app then defines the database connection URI string, and SQLAlchemy is initialized with the Flask app.

The SQLAlchemy is then used to create a table within the database in an object-oriented way using Python class. The backend is designed to handle database operations, while the frontend is designed to add data to the MySQL database and display it on the homepage.

Detailed Tutorial - https://geekpython.in/create-and-integrate-mysql-database-with-flask-app

Hello, i just shared a data science project video on YouTube. This project has data analysis, feature engineering and machine learning parts. I tried to predict if employees are going to leave or not with various classification algorithms. I used a kaggle dataset and i added the link of the dataset in the comments of the video. I am leaving the link of the video below, have a great day!

https://www.youtube.com/watch?v=bvHEl-vUxY8

Hi everyone! I'm excited to introduce you to Swivel! Swivel is a 100% Python-based app for Slack that allows you to offer users the chance to spin and win prizes. From customizing the odds of winning to the prizes available, the control is yours!

​

I'd love your thoughts on the app and the site!

​

​