df2 = pd.DataFrame([
[-0.532681, 'foo', 0],
[1.490752, 'bar', 1],
[-1.387326, 'foo', 2],
[0.814772, 'baz', ' '],
[-0.222552, ' ', 4],
[-1.176781, 'qux', ' '],
],columns=['one', 'two', 'three'])
# df2
df2['two'] = df2['two'].replace(r'^\s*$', np.nan, regex=True)
df2
If we have to do the same in entire dataframe, just do this below
df2 = df2.replace(r'^\s*$', np.nan, regex=True)
Using the host : guest format you can do any of the following:
volumes:
# Just specify a path and let the Engine create a volume
- /var/lib/mysql
# Specify an absolute path mapping
- /opt/data:/var/lib/mysql
# Path on the host, relative to the Compose file
- ./cache:/tmp/cache
# User-relative path
- ~/configs:/etc/configs/:ro
# Named volume
- datavolume:/var/lib/mysql
Long Syntax
As of docker-compose v3.2 you can use long syntax which allows the configuration of additional fields that can be expressed in the short form such as mount type (volume, bind or tmpfs) and read_only.
version: "3.2"
services:
web:
image: nginx:alpine
ports:
- "80:80"
volumes:
- type: volume
source: mydata
target: /data
volume:
nocopy: true
- type: bind
source: ./static
target: /opt/app/static
networks:
webnet:
volumes:
mydata:
references:
https://stackoverflow.com/questions/40905761/how-do-i-mount-a-host-directory-as-a-volume-in-docker-compose
#below gives values counts on the name column. If there are many empties in the column, then it gives that count
print(data['name'].value_counts())
Below are some variants.
data['marks'].value_counts(ascending=False)
data['age'].describe()
Group by gives values in a column group by
data.groupby('subjects').size()
JSON file can sometimes be clumsy having different levels and hierarchy. Pandas have a nice inbuilt function called json_normalize() to flatten the simple to moderately semi-structured nested JSON structures to flat tables.
Below are different ways Pandas provide to convert JSON to dataframe
# Use json_normalize() to convert JSON to DataFrame
dict= json.loads(data)
df = json_normalize(dict['technologies'])
# Convert JSON to DataFrame Using read_json()
df2 = pd.read_json(jsonStr, orient ='index')
# Use pandas.DataFrame.from_dict() to Convert JSON to DataFrame
dict= json.loads(data)
df2 = pd.DataFrame.from_dict(dict, orient="index")
If the JSON is like below
{
'_index': 'complaint-public-v2',
'_type': 'complaint',
'_id': '3230997',
'_score': 0.0,
'_source': {'tags': None,
'zip_code': '49508',
'complaint_id': '3230997',
'issue': 'Managing an account',
'date_received': '2019-05-01T12:00:00-05:00',
'state': 'MI',
'consumer_disputed': 'N/A',
'product': 'Checking or savings account',
'company_response': 'Closed with monetary relief',
'company': 'JPMORGAN CHASE & CO.',
'submitted_via': 'Referral',
'date_sent_to_company': '2019-05-02T12:00:00-05:00',
'company_public_response': None,
'sub_product': 'Checking account',
'timely': 'Yes',
'complaint_what_happened': '',
'sub_issue': 'Problem making or receiving payments',
'consumer_consent_provided': 'N/A'}
}
df = pd.json_normalize(test_dict)
df.columns
Index(['_index', '_type', '_id', '_score', '_source.tags', '_source.zip_code',
'_source.complaint_id', '_source.issue', '_source.date_received',
'_source.state', '_source.consumer_disputed', '_source.product',
'_source.company_response', '_source.company', '_source.submitted_via',
'_source.date_sent_to_company', '_source.company_public_response',
'_source.sub_product', '_source.timely',
'_source.complaint_what_happened', '_source.sub_issue',
'_source.consumer_consent_provided'],
dtype='object')
When load data frame using json_normalize, it becomes like this below
references:
https://www.geeksforgeeks.org/python-pandas-flatten-nested-json/
Usually done through LDA(Latent Dirichlet Allocation). It identifies topics that describes a document or set of documents.
The word latent is because the topics will only evolve during the modelling process. Topic modelling is an unsupervised task.
This is mainly done by identifying the patterns of word clusters and frequencies of words in the document.
LDA short summary (Latent Dirichlet Allocation)
Dirichlet is form of distribution, which is different from Normal distribution. The ML algorithms can be applied where the data is normally distributed and it works with real numbers. In Dirichlet, the plotted data sum up to 1. Dirichlet is a probability distribution that is sampling over probability simplex instead of sampling from the space of real numbers as in Normal distribution.
LDA brings the words in the topic with their distribution using Dichrlet distribution. The words assigned to the topic with their distribution using Dichrlet distribution.
References:
https://www.analyticsvidhya.com/blog/2021/05/topic-modelling-in-natural-language-processing/
From a corpus of words, a word is converted to its base form . Eg: fix, fixing, fixed gives fix. Different types of stemming are
1. Porter Stemmer,
2. Lancaster Stemmer,
3. Snowball Stemmer
Lemmatization cuts the word to gets its lemma word meaning it gets a much more meaningful form than what stemming does. The output we get after Lemmatization is called ‘lemma’.
For e.g. Having is converted to Hav in Stemming, while Lemmatization converts it to Have.
Some of them are WordNet Lemmatization, TextBlob, Spacy, Tree Tagger, Pattern, Genism, and Stanford CoreNLP lemmatization.
references:
https://www.analyticsvidhya.com/blog/2021/05/topic-modelling-in-natural-language-processing/
Time duration
A duration is defined as a ISO 8601 durations format, which defines the amount of intervening time in a time interval and are represented by the format P(n)Y(n)M(n)DT(n)H(n)M(n)S. Note that the n is replaced by the value for each of the date and time elements that follow the n.
The capital letters P, Y, M, W, D, T, H, M, and S are designators for each of the date and time elements and are not replaced, but can be omitted.
P is the duration designator (for period) placed at the start of the duration representation.
Y is the year designator that follows the value for the number of years.
M is the month designator that follows the value for the number of months.
W is the week designator that follows the value for the number of weeks.
D is the day designator that follows the value for the number of days.
T is the time designator that precedes the time components of the representation.
H is the hour designator that follows the value for the number of hours.
M is the minute designator that follows the value for the number of minutes.
S is the second designator that follows the value for the number of seconds.
Examples:
PT15S - 15 seconds
PT1H30M - 1 hour and 30 minutes
P14D - 14 days
P14DT1H30M - 14 days, 1 hour and 30 minutes
P3Y6M4DT12H30M5S - 3 years, 6 months, 4 days, 12 hours, 30 minutes and 5 seconds
If the duration is zero or negative, the timer fires immediately.
references:
https://docs.camunda.io/docs/components/modeler/bpmn/timer-events/
